CAPTURING  THE  POLICY  THAT  AIR  FORCE  RATERS  USE  WHEN  WRITING 
PERFORMANCE  APPRAISALS  ON  JUNIOR  OFFICERS 

THESIS 


Owen  D.  Stephens,  First  Lieutenant,  USAF 


AFIT/GAQ/ENV/0 1 M- 1 3 

DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 


Wright-Patterson  Air  Force  Base,  Ohio 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


20010612  U9 


The  views  expressed  in  this  thesis  are  those  of  the  author  and  do  not  reflect  the  official 
policy  or  position  of  the  United  States  Air  Force,  Department  of  Defense,  or  the  U.  S. 
Government. 


AFIT/GAQ/ENV/0 1 M- 1 3 


CAPTURING  THE  POLICY  THAT  AIR  FORCE  RATERS  USE  WHEN  WRITING 
PERFORMANCE  APPRAISALS  ON  JUNIOR  OFFICERS 


THESIS 


Presented  to  the  Faculty 

Department  of  Systems  and  Engineering  Management 
Graduate  School  of  Engineering  and  Management 
Air  Force  Institute  of  Technology 
Air  University 

Air  Education  and  Training  Command 
In  Partial  Fulfillment  of  the  Requirements  for  the 
Degree  of  Master  of  Science  in  Acquisition  Management 


Owen  D.  Stephens,  B.S. 
First  Lieutenant,  USAF 

March  2001 


APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  UNLIMITED. 


AFIT/GAQ/ENV/0 1 M- 1 3 


CAPTURING  THE  POLICY  THAT  AIR  FORCE  RATERS  USE  WHEN  WRITING 
PERFORMANCE  APPRAISALS  ON  JUNIOR  OFFICERS 

Owen  D.  Stephens,  B.S. 

First  Lieutenant,  USAF 

Approved: 


Acknowledgments 


I  would  like  to  express  my  sincere  appreciation  to  my  advisor,  Major  Michael 
Rehg,  for  his  efforts  to  help  me  tackle  this  difficult  topic.  I  struggled  with  many  of  the 
concepts  in  this  research  effort  and  he  always  made  himself  available  to  help  me  with  my 
difficulties.  I  would  also  like  to  thank  Major  Paul  Thurston.  I  could  not  have  completed 
this  project  without  his  understanding  of  policy  capturing  and  the  time  he  took  to  teach 
me.  Colonel  Joseph  Amend  and  Lieutenant  Colonel  David  Petrillo  also  gave  their  time 
and  talents  to  this  effort,  and  their  contributions  are  much  appreciated.  I  would  also  like 
to  thank  Lieutenant  Colonel  (Retired)  James  Van  Scotter  for  his  contributions. 

In  spite  of  the  efforts  of  my  advisor  and  readers,  I  could  not  have  finished  this 
thesis  without  the  help  of  my  wife  and  daughter.  My  daughter’s  help  was  indirect 
but  still  valuable.  She  was  understanding  of  the  amount  of  work  that  I  had  to 
do  and  took  great  care  let  me  work  uninterrupted.  My  wife’s  help  was  direct.  When  I 
broke  my  collarbone  and  could  not  drive  or  type,  she  drove  me  to  school  and  typed  my 
work.  When  I  needed  help  entering  data  because  the  pain  from  my  injury  and  subsequent 
surgery  hindered  me,  she  took  over  and  completed  the  task.  She  was  my  most  outspoken 
critic  and  toughest  editor.  If  my  thesis  were  dual  authored,  it  would  be  my  wife’s  name 
beside  mine  on  the  title  page.  I  am  truly  lucky  to  have  both  by  my  side. 

Finally,  but  most  importantly,  I  would  like  to  thank  God  for  the  blessings  he  has 
given  me.  Only  through  His  grace  can  I  accomplish  anything. 

Owen  D.  Stephens 


IV 


Table  of  Contents 


Page 


Acknowledgments . iv 

List  of  Figures . viii 

List  of  Tables . ix 

Abstract . xi 

I.  Introduction . 1 

General  Issue . 1 

Research  Questions . 5 

Statement  of  Problem . 6 

Scope  of  Research . 7 

II.  Literature  Review . 8 

Introduction . 8 

The  Rating  Approach  Used . 8 

Rater  Priority  and  Policy  Capturing . 10 

Gender  Effects . 11 

Demographics . 11 

Rater  Commissioning  Source  and  Experience . . 12 

Mood  Traits  and  Personality . 1 3 

The  Four  Dimensional  Overall  Performance  Model  and  Associated  Behaviors . 16 

Leadership . 19 

Task  Performance . 19 


v 


Page 

Interpersonal  Facilitation . 20 

Job  Dedication . 21 

The  Foundation  of  This  Research . 22 

Objectives  and  Hypothesis . 23 

III.  Method . 27 

The  Survey  Instrument . 27 

Pilot  Survey . 27 

Sample  and  Procedure . 28 

Survey  Instrument . 28 

Final  Survey . 32 

Suggestions  and/or  Comments  Received  on  the  Pilot  Survey . 32 

Changes  Made  for  the  Final  Survey . 34 

Sample  and  Procedure . 36 

Analysis . 37 

IV.  Results . 40 

Demographic  Information . 40 

Personality  Items  and  PANAS— Correlation,  Reliability,  and  Factor  Analyses . 41 

The  First  Level  Regression  of  Each  Respondent’s  Profiles . 44 

Results  for  the  Individual  Behavior  Ratings . 45 

Results  for  Differences  in  the  Relative  Importance  of  Performance  Dimensions . 52 

Second  Level  Regression . 56 


vi 


Page 


Consistency  Results . 57 

Leniency  Results . 58 

Ratee  Gender  Effects  Results . 59 

V.  Conclusions  and  Discussions . 60 

Relative  Importance  of  Behavior  Dimensions  to  Overall  Performance . 60 

Individual  Behavior  Means . 62 


Second  Level  Regression  With  Demographic,  Personality,  and  PANAS  Variables.... 63 


Rater  Consistency . 64 

Rating  Leniency . 65 

Ratee  Gender  Effects . 66 

Weaknesses . 66 

Future  Research . 69 

Appendix  A.  Leadership  Behaviors . 71 

Appendix  B.  Task  Performance  Behaviors . 72 

Appendix  C.  Interpersonal  Facilitation  Behaviors . 73 

Appendix  D.  Job  Dedication  Behaviors . 74 

Appendix  E.  Beta  Weights . 75 

Appendix  F.  Survey . 78 

Bibliography . 94 

Vita . 99 


vii 


List  of  Figures 


Page 

Figure  1.  The  Model  of  Air  Force  Rater  Priorities . 8 

Figure  2.  Four-Factor  Model  of  Junior  Officer  Performance  (Flurry,  1 995) . 22 

Figure  3.  Survey  Scenario . 29 


viii 


List  of  Tables 


Page 

Table  1.  Behavior  Consolidations  for  Profile  Construction . 30 

Table  2.  Pilot  Between  Profile  Correlation  of  Behavior  Dimensions  (high  correlation).  35 

Table  3.  Final  Survey  Between  Profile  Correlation  of  Behavior  Dimensions . 35 

Table  4.  Survey  Type . 40 

Table  5.  Rank  of  Respondents . 40 

Table  6.  Gender  of  Respondents . 40 

Table  7.  Job  Category  of  Respondents . 41 

Table  8.  Commissioning  Source  of  Respondents . 41 

Table  9.  Reliability  Analysis  Results  for  Conscientiousness  Factor . 42 

Table  10.  Factor  Analysis  Varimax  Rotation . 43 

Table  1 1 .  Factor  Analysis  Manual  Extraction . 43 

Table  12.  Factor  Analysis  PANAS  Scales,  Manual  Extraction . 44 

Table  13.  Descriptive  Statistics  for  the  Individual  Behavior  Dimension  Ratings . 46 

Table  14.  Leadership  Behavior  Means . : . 47 

Table  15.  Task  Performance  Behavior  Means . 48 

Table  16.  Interpersonal  Facilitation  Behavior  Means . 49 

Table  17.  Job  Dedication  Behavior  Means . 50 

Table  18.  Means  Comparison  Summary  Table . 51 

Table  19.  Correlation  Table  of  Means . 51 


IX 


Page 


Table  20.  Final  Survey  Sub  Sample  Comparison  of  Means . 52 

Table  21 .  Sub  Sample  Comparison  Correlation  Table . 52 

Table  22.  Descriptive  Statistics  of  the  Four  Dimensions’  Beta  Weights  from  Regression 

on  Overall  Performance . 53 

Table  23.  Mean  Beta  Weights  for  Behavior  Dimensions  of  Sample  and  Various  Sub 

Samples  Derived  from  Regression  on  Overall  Performance . 54 

Table  24.  Test  of  Normality . 55 

Table  25.  Tukey-Kramer  HSD  Means  Comparisons . 56 

Table  26.  Model  Summary  for  Consistency  Regression . 58 

Table  27.  ANOVA  Table  for  Consistency  Regression . 58 

Table  28.  Coefficients  Table  for  Consistency  Regression . 58 

Table  29.  Significant  Correlations  Between  Average  Overall  Performance  and 

Personality  Measures . 59 

Table  30.  (Modified  from  Hurry,  1995:  16) — Leadership  Behaviors . 71 

Table  31.  (Modified  from  Hurry  1995:  18) — Task  Performance  Behaviors . 72 

Table  32.  (Modified  from  Hurry  1995:  19) — Interpersonal  Facilitation  Behaviors . 73 

Table  33.  (Modified  from  Hurry,  1995:  21) — Job  Dedication  Behaviors . 74 

Table  34.  Beta  Weights  and  R  Values  (n=48  for  every  regression) . 75 


x 


AFIT/GAQ/ENV/0 1 M- 1 3 


Abstract 

This  study  examined  the  relationship  between  four  dimensions  of  performance — 
leadership,  task  performance,  interpersonal  facilitation,  and  job  dedication — and  overall 
performance  by  junior  officers  in  the  Air  Force.  It  was  hypothesized  that  the  four 
dimensions  would  have  different  relative  importance  to  overall  performance,  ratee  gender 
would  have  no  effect  on  rating  policy,  and  raters  with  greater  experience  would  show 
greater  rating  consistency.  Finally,  it  was  hypothesized  that  the  Conscientiousness  factor 
of  the  International  Personality  Inventory  Protocol  and  Positive  and  Negative  Affectivity 
may  be  correlated  with  the  relative  importance  of  the  four  dimensions  of  performance 
and  a  rater  leniency  measure  and  that  significant  prediction  models  could  be  developed. 

For  the  entire  sample  and  a  sub  sample  of  Majors-and-above,  leadership  was  the 
most  important  dimension.  The  other  dimensions — task  performance,  interpersonal 
facilitation,  and  job  dedication — were  equally  important,  yet  less  important  than 
leadership.  For  Captains  and  below,  all  dimensions  were  equally  important.  There  was 
no  significant  gender  effect  on  ratings,  but  there  was  evidence  that  experience  is 
positively  related  to  consistency  of  ratings  for  Majors  and  above.  Neither 
Conscientiousness  nor  Positive  and  Negative  Affectivity  could  be  used  to  develop  a 
significant  prediction  model  for  the  importance  of  behavior  dimensions  or  rater  leniency. 
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CAPTURING  THE  POLICY  THAT  AIR  FORCE  RATERS  USE  WHEN  WRITING 


PERFORMANCE  APPRAISALS  ON  JUNIOR  OFFICERS 


I.  Introduction 


General  Issue 

The  Air  Force  has  changed  dramatically  in  recent  years  and  the  expectations  that 
raters  have  for  junior  officers  may  have  changed  with  it.  As  roles  the  Air  Force  must 
play  adapt  to  the  changing  world  environment,  the  roles  of  junior  officers  must  also 
adapt.  What  are  the  current  roles  of  the  junior  officer  as  measured  by  the  expectations  of 
the  officers  who  rate  them?  This  is  one  of  the  primary  questions  this  research  hopes  to 
answer  by  measuring  the  rating  bias  of  officers  who  are  in  a  position  to  rate  junior 
officers. 

Raters  have  two  levels  of  bias  that  they  must  work  through  to  evaluate  a  junior 
officer.  The  first  bias  deals  with  the  individual  behaviors  of  the  junior  officer  that  the 
rater  observes.  What  types  of  behaviors  are  most  important  for  the  success  of  junior 
officers?  This  question  has  been  answered  for  a  sample  of  possible  behaviors  and  is 
discussed  later  in  this  thesis.  Each  of  these  behaviors  has  an  importance  weighting  that 


1 


varies  slightly  from  rater  to  rater.  For  example,  one  rater  may  weigh  “gets  along  with 
others”  significantly  less  than  “cooperates  with  others  on  the  team  effectively.” 

The  second  level  of  bias  deals  with  dimensions  of  behaviors.  This  research  effort 
divided  the  behavior  dimensions  into  four  broad  categories:  leadership,  task  performance, 
interpersonal  facilitation,  and  job  dedication.  The  rater  may  display  a  bias  toward  or 
against  a  particular  behavior  dimension.  This  bias  will  be  displayed  in  the  weighting  the 
rater  uses  for  each  dimension  to  decide  the  junior  officer’s  overall  performance.  For 
example,  a  rater  may  feel  that  a  junior  officer  has  displayed  very  good  interpersonal 
facilitation  behaviors,  but  the  interpersonal  facilitation  dimension  may  not  be  important 
to  that  rater  and  so  the  rater  does  not  weight  this  facet  of  performance  heavily  when 
rating  the  overall  effectiveness  of  the  junior  officer. 

Junior  officers  initially  learn  how  they  will  be  rated  while  attending  their 
commissioning  source  program.  After  they  enter  the  Air  Force,  peers,  supervisors,  and 
Professional  Military  Education  (PME)  reinforce  their  knowledge.  Therefore,  there  are 
four  primary  sources  where  junior  officers  learn  about  officer  effectiveness  and 
appropriate  behaviors:  commissioning  programs,  PME,  supervisors,  and  peers. 

The  three  commissioning  sources — U.S.  Air  Force  Academy  (USAFA),  Reserve 
Officer  Training  Corp  (ROTC),  and  Officer  Training  School  (OTS)— provide  officers 
basic  knowledge  of  the  behaviors  that  will  make  them  successful.  Individuals  in  these 
commissioning  sources  receive  instruction  in  communication  skills,  leadership,  and 
managerial  practices  expected  to  be  important  in  their  Air  Force  roles  (Air  Command  and 
Staff  College,  1988:  10). 


2 


After  entering  the  Air  Force,  junior  officers  receive  further  instruction  at  PME 
schools.  Squadron  Officer  School  (SOS),  the  primary  junior  officer  PME,  reinforces  the 
same  knowledge  that  the  officers  learned  at  their  commissioning  sources.  Some  bases 
also  have  an  informal,  local  PME  called  Lieutenants  Professional  Development  Program 
(LPDP).  An  examination  of  this  curriculum  shows  this  course  is  a  scaled-down  version 
of  the  curriculum  at  SOS  (Company  Grade  Officer  Professional  Development  Program 
Study  Guide,  1999). 

Junior  officers  also  receive  information  about  important  behaviors  from  their 
supervisors— referred  to  in  the  rest  of  this  document  as  raters.  The  Officer  Evaluation 
System  is  the  avenue  for  formal  education  about  expectations  for  junior  officers.  This 
system  appraises  individual  performance  and  provides  an  avenue  for  feedback  concerning 
the  individual’s  on-going  performance.  Some  raters  may  also  form  a  mentoring 
relationship  with  their  junior  officers  and  provide  them  with  informal  insight  into  the 
behaviors  that  are  expected  of  them. 

Finally,  peer  networks  provide  an  informal  source  of  knowledge  about  successful 
and  unsuccessful  behaviors.  Junior  officers  may  gather  at  the  Officer’s  Club  after  work 
or  meet  at  Company  Grade  Officer’s  Council  meetings.  Here,  the  junior  officers  share 
their  own  personal  experiences  about  which  behaviors  have  afforded  them  success.  They 
may  also  share  their  experience  about  which  behaviors  have  damaged  their  career 
progress. 

Although  there  is  plenteous  research  concerning  performance  appraisals  and 
behaviors,  little  empirical  research  has  been  directed  at  discovering  which  types  of 
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behaviors  and  behavior  dimensions  are  important  to  Air  Force  raters.  Recently,  however, 
the  three  commissioning  sources  began  to  re-analyze  their  key  processes,  programs,  and 
customer  requirements  (Hurry,  1995:  2).  The  commissioning  sources  needed  to  ensure 
that  graduates  have  the  specific  knowledge,  skills  and  behaviors  needed  to  be  successful 
in  their  Air  Force  careers,  but  there  was  little  empirical  evidence  to  use  in  their  analysis 
(Hurry,  1995:  2).  In  1994,  a  study  was  performed  at  the  Air  Force  Institute  of 
Technology  (AFIT)  to  capture  the  policy  that  Air  Force  raters  use  when  appraising  their 
subordinates  (Hurry,  1995).  This  study  developed  a  good  model  for  identifying  which 
behaviors  are  important  to  Air  Force  raters,  but  a  methodological  error  may  have  caused 
the  results,  and  therefore  the  conclusions  drawn  from  them,  to  be  invalid.  The  researcher 
failed  to  randomize  the  behavior  order  within  the  fictitious  profiles.  The  resulting  order 
of  behavior  dimension  importance  mirrored  the  order  of  the  behavior  dimensions  in  the 
profiles.  This  will  be  discussed  in  greater  detail  in  Chapter  II. 

This  thesis  will  readdress  these  issues  by  proposing  and  testing  a  comprehensive 
set  of  behaviors  and  identifying  the  most  important  behaviors  for  successful  performance. 
This  thesis  will  also  overcome  any  potential  problem  caused  by  the  order  effect  present  in 
the  previous  research  and  attempt  to  validate  or  invalidate  the  previous  research  by 
identifying  the  order  of  importance  of  the  four  behavior  dimensions  to  Air  Force  raters. 

This  thesis  will  add  to  the  previous  research  by  measuring  raters’  personality  traits 
and  identifying  how  these  traits  may  influence  performance  appraisal  policy.  For 
example,  a  high  level  of  the  personality  trait  of  “Conscientiousness,”  may  be  present  in 
raters  who  display  more  consistency  in  their  assessment  of  junior  officers.  The  traits  this 
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research  will  explore  are  the  six  facets  of  the  personality  factor  Conscientiousness — Self- 
Efficacy,  Orderliness,  Dutifulness,  Achievement-Striving,  Self-Discipline,  and 
Cautiousness  (Goldberg,  1999).  It  will  also  explore  mood  traits  with  a  brief  measure  of 
positive  and  negative  affectivity  (PANAS)  (Watson,  Clark,  and  Tellegen,  1988).  The 
results  of  this  research  can  help  guide  the  commissioning  sources  in  their  efforts  to  build 
curriculum  that  teaches  students  the  appropriate  knowledge,  skills,  and  behaviors  needed 
in  a  successful  Air  Force  career. 

Research  Questions 

The  primary  purpose  of  this  thesis  is  to  determine  the  knowledge,  skills,  and 
behaviors  that  raters  feel  contribute  the  most  to  the  overall  performance  of  junior  officers. 
Since  an  officer’s  overall  performance  is  a  product  of  his  or  her  performance  in  a  variety 
of  tasks  (Slovic  and  Lichtenstein,  1977),  this  study  will  use  the  policy  capturing  method 
to  determine  how  the  raters  use  the  dimensions  to  arrive  at  an  overall  rating.  This  method 
measures  the  perceived  importance  of  each  chosen  behavior  category  and  to  what  degree 
that  category  influences  the  overall  rating  given  to  the  junior  officers. 

The  second  objective  of  this  study  is  to  explore  the  possibility  that  other 
demographic  factors  of  raters  may  influence  their  appraisal  policy.  Therefore  the  study 
will  examine  the  influence  of  experience,  commissioning  source,  and  gender  on  raters’ 
ratings.  Experience  will  be  examined  using  several  measures:  rank,  total  time  in  service, 
and  past  experience  at  rating. 
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The  third  objective  of  this  study  is  to  determine  if  rater  personality  traits  affect  the 
importance  they  place  on  behavior  dimensions,  the  ratings  they  give  to  subordinates,  or 
the  amount  of  rating  consistency  or  leniency  that  they  display.  Mood,  captured  as  a  long¬ 
term  trait,  is  one  important  trait  that  may  affect  raters’  policy.  Therefore,  this  study  will 
capitalize  on  previous  research  that  provides  valid  measures  of  two  dimensions  of  mood, 
positive  and  negative  affect  (Watson,  Clark,  and  Tellegen,  1988).  This  study  will  also 
measure  six  facets  of  personality,  as  previously  mentioned,  to  determine  if  they  influence 
raters’  ratings  (Goldberg,  1999). 

The  fourth  objective  of  this  study  is  to  determine  if  rater  career  fields  have  any 
bearing  on  their  appraisal  policies.  This  research  will  divide  military  raters  into  four 
major  categories:  rated  officers,  support  officers/managers,  analysts/engineers,  and 
others.  These  categories  were  tested  in  previous  research  (Hurry,  1 995)  and  no 
significant  differences  were  found  between  job  categories.  However,  order  effects  may 
have  confounded  the  research,  so  this  study  will  attempt  to  validate  the  previous  findings. 

The  final  objective  of  this  study  is  to  determine  if  the  gender  of  the  subordinate 
affects  raters’  rating  policies.  This  study  will  use  a  visual  cue — the  name  of  the  ratee — to 
identify  the  gender  of  the  rated  junior  officer  and  examine  if  this  cue  has  any  influence  on 
the  ratings. 

Statement  of  Problem 

The  purpose  of  this  research  is  to  identify  the  weight  that  raters  apply  to  behavior 
dimensions  when  they  complete  the  ratings  of  hypothetical  profiles  of  junior  officers. 
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The  research  will  also  identify  other  factors  that  may  bear  on  raters’  decisions  when  they 
complete  performance  appraisals. 

Scope  of  Research 

This  study  is  limited  to  the  behaviors  that  are  important  for  junior  officers.  Junior 
officer  is  defined  here  as  a  Company  Grade  Officer  (CGO) — 2nd  Lieutenant  through 
Captain.  The  range  this  research  will  consider  for  raters  of  junior  officers  is  2nd 
Lieutenant  through  Lieutenant  Colonel.  Generally,  only  Captains  and  above  have 
experience  rating  junior  officers,  but  it  is  important  to  be  able  to  contrast  perceptions  of 
junior  officers  to  those  of  higher  rank.  Finally,  this  study  will  be  limited  to  four  Air 
Force  bases  and  a  sample  of  convenience  of  Air  Force  Institute  of  Technology  (AFIT) 
and  Defense  Acquisition  University  (DAU)  students  due  to  the  limited  resources  and  time 
of  the  researcher. 
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II.  Literature  Review 


Introduction 

The  model  in  Figure  1  provides  a  pictorial  representation  of  this  research.  This 
thesis  will  develop  the  model,  discussing  each  of  the  constructs  throughout  this  section  by 
examining  previous  research  that  provides  background  information. 


Figure  1.  The  Model  of  Air  Force  Rater  Priorities 

The  Rating  Approach  Used 

The  process  approach  to  assessment  was  used  for  this  research  effort  out  of  three 
possible  alternatives  presented  by  Campbell,  Dunnette,  Lawler,  and  Weick —  the  product 
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approach,  the  person  approach,  and  the  process  approach  (Campbell  et  al.,  1970:  1 75). 
The  product  approach  was  infeasible  because  of  the  methodology  chosen.  It  measures 
physical  output  and  that  is  not  possible  using  the  fictitious  profile  survey  method, 
especially  since  product  is  job  specific  and  this  method  dealt  in  the  realm  of  generalities. 

The  person  approach  revolves  around  the  theory  that  individuals  have  traits  or 
characteristics.  Some  of  these  traits  would  make  individuals  effective  performers  while 
others  make  individuals  ineffective  performers.  First,  traits  would  be  difficult  to  present 
using  the  method  chosen  for  this  research.  Also,  traits  may  not  necessarily  result  in 
effective  performance  because  the  traits  are  not  backed  up  by  behaviors.  In  order  to  truly 
result  in  effective  performance,  the  traits  would  have  to  be  followed  by  appropriate 
actions. 

The  process  approach  examines  performance  using  the  behaviors  exhibited  by  the 
individual  being  rated.  This  approach  lends  itself  to  the  fictitious  profile  survey  method 
because  behaviors  can  be  seen  and  recorded.  In  fact,  Officer  Performance  Reports  rely 
heavily  on  reporting  individual  behaviors.  Kirkpatrick  and  Locke  (1991 :  501)  wrote  that 
behaviors  and  product  can  be  reliably  observed,  but  traits  cannot  be  reliably  observed. 
Also,  there  is  an  ample  amount  of  research  dedicated  to  behaviors,  providing  this  effort 
with  an  adequate  supply  of  positive  behavior  for  each  of  the  four  performance 
dimensions.  Therefore,  of  the  three  approaches,  the  process  approach  was  the  best  choice 
given  the  method  used  to  gather  rater  policy. 
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Rater  Priority  and  Policy  Capturing 


The  organizational  norms  concerning  rater  priority  are  the  main  area  of  interest. 
This  cannot  be  measured  directly,  so  it  must  be  measured  through  the  constructs  of 
“Rater  Priority”  and  “Rating  Decisions”.  The  only  construct  that  can  be  measured 
directly  is  “Rating  Decisions,”  and  it  will  be  measured  using  the  profile  scenario  method 
of  policy  capturing. 

The  term  "policy"  is  defined  as  "the  factors  used  in  making  a  judgment  and  the 
relative  weighting  thereof'  (Ullman  and  Doherty,  1984:  179).  Within  this  same  context, 
"policy  capturing"  refers  to  "studies  that  analyze  judgments  made  on  the  basis  of 
multidimensional  stimuli  by  means  of  a  linear  model"  (Brehmer  and  Brehmer,  1988:  78). 
A  growing  body  of  research  is  using  policy  capturing  as  a  method  of  addressing  rater 
decision-making  in  performance  appraisals  (Hobson  and  Gibson,  1983).  The  use  of 
policy  capturing  is  compatible  with  the  efforts  of  this  study.  This  method  measures  the 
perceived  importance  of  each  chosen  behavior  category  and  to  what  degree  that  category 
influences  the  overall  rating  given  to  the  junior  officers.  Policy  capturing  will  provide 
the  means  to  model  the  relationship  between  the  chosen  behavior  dimensions  and  the 
supervisor’s  overall  performance  rating  policies. 

This  study  will  use  self-reporting  to  capture  the  policy  that  raters  use  when 
making  an  assessment.  Unfortunately,  several  problems  exist  with  asking  individuals  to 
report  on  the  factors  that  affect  their  own  decisions  and  judgments.  Research  has  shown 
that  people  typically  do  not  have  good  insight  into  the  various  influences  involved  in  their 
decision-making  processes,  even  though  respondents  typically  believe  that  they  do 
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(Brehmer  and  Brehmer,  1988;  Nisbett  and  Ross,  1980;  Nisbett  and  Wilson,  1977).  This 
same  problem  has  also  been  evidenced  concerning  individuals’  rating  policies  (Hobson  et 
al.,  1981 ;  Taylor  and  Wilsted,  1974;  Zedeck  and  Kafry,  1977).  In  order  to  overcome  this 
problem,  a  method  of  analysis  that  measures  the  importance  of  the  dimensions  of 
performance  without  capturing  the  poor  insight  is  needed  (Hobson  and  Gibson,  1983). 
Hobson  and  Gibson  (1983)  suggest  that  field  studies  would  be  the  most  effective  method 
to  achieve  this,  but  self-reporting  on  hypothetical  profiles  is  necessary  to  generate  the 
advantage  of  a  large  number  of  profiles. 

Gender  Effects 

A  growing  body  of  literature  in  the  field  of  gender  effects  on  performance 
appraisals  has  produced  mixed  results.  Empirical  evidence  of  performance  appraisals 
does  not  wholly  support  the  popular  news  media  notion  that  women  are  treated  unfairly. 
While  studying  women  in  the  accounting  career  field,  Fogarty,  Parker,  and  Robinson 
(1998)  provided  empirical  evidence  that  females  do  not  receive  lower  ratings  than  males. 
Other  researchers  (Dobbins  and  Trahan,  1986;  Shore  and  Thornton,  1982)  have  examined 
this  area  and  found  that  women  receive  higher  ratings  than  men.  Lewis  (1997)  studied 
civilians  in  the  federal  service  and  also  found  that,  in  general,  women  receive  higher 
ratings  than  men  in  all  job  categories  and  every  grade  level. 

Demographics 

Rater  demographics  are  the  variables  such  as  rater  sex  and  Air  Force  job  category. 
Hurry  (1995)  found  that  these  variables  explained  none  of  the  variance  in  behavior 
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ratings  or  overall  performance  ratings.  The  effect  of  these  variables  may  have  been 
confounded  by  the  order  effects  present  in  her  study  and  should  be  examined  again. 

There  were  three  job  categories  chosen  for  this  research — rated  officers,  support 
officers,  and  analysts/engineers.  There  was  also  an  “other”  category  for  officers  that  do 
not  neatly  fit  into  one  of  these  three  categories.  The  rated  officer  category  includes  pilots 
and  navigators.  These  officers  generally  are  not  given  supervisory  positions  until  they 
have  been  promoted  beyond  the  junior  officer  ranks.  Support  officers  form  the  second 
category  being  examined  and  include  logisticians,  administrative/executive  officers, 
personnel  officers,  acquisition  officers,  civil  engineers,  etc.  It  would  not  be  unusual  for 
support  officers  to  be  placed  in  supervisory/leadership  positions  early  in  their  careers. 
Finally,  analysts/engineers  make  up  the  last  job  category  being  examined.  The  category 
includes  operations  researchers,  design  engineers,  aeronautical  engineers,  and  other 
technical  jobs  as  would  be  found  in  the  Air  Force  Research  Labs.  Analysts/engineers 
usually  have  very  little  opportunity  to  supervise  or  lead  subordinate  Air  Force  personnel. 
These  categories  are  not  the  only  categories  in  the  Air  Force,  but  do  represent  the 
majority  of  officers.  There  may  be  a  large  enough  response  in  the  “other”  category, 
however,  to  justify  inclusion  of  an  additional  category  in  the  analysis. 

Rater  Commissioning  Source  and  Experience 

Borman  et  al.  (1987)  theorized  and  Van  Scotter  and  Shane  (1995)  found  that  job 
experience  could  affect  rating  judgments  of  supervisors.  Van  Scotter  and  Shane  ( 1 995) 
reported,  “the  source  of  entry-level  training  [commissioning  source]  also  made  a 
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difference,  but  it  is  confounded  here  by  a  relationship  with  the  experience  variable”  (Van 
Scotter  and  Shane,  1995:  224).  Because  previous  research  suggests  that  rater  experience 
and  commissioning  source  may  influence  rating  judgment,  these  two  variables  will  be 
examined. 

In  the  Air  Force,  grade  level  and  experience  are  highly  correlated.  One  could 
expect  that  with  experience  comes  consistency,  and  therefore  the  more  experienced 
officers  would  tend  to  rate  behaviors  more  consistently.  R  provides  an  adequate 
measure  of  rating  consistency  in  policy  capturing  studies  because  more  consistent 
respondent  ratings  result  in  a  better  model  fit  and  therefore  a  higher  R  (Hobson  and 
Gibson,  1 983).  R2,  a  sample  statistic,  implies  that  it  can  be  used  as  an  indicator  of  the 
usefulness  of  the  entire  model  for  predicting  (McClave,  Benson,  and  Sincich,  1998). 
Therefore,  a  high  R2  indicates  that  the  regression  model  is  an  appropriate  tool  to  use  for 
capturing  raters’  priorities. 

Mood  Traits  and  Personality 

Finally,  this  study  will  examine  the  effects  of  rater  personality  on  a  rater’s  overall 
rating  policy.  Research  has  suggested  that  individual  personality  difference  variables  can 
make  a  substantive  contribution  to  the  selection  equation  when  trying  to  predict  job 
performance  (Day  and  Silverman,  1989;  Hough,  Eaton,  Dunnette,  Kamp,  and  McCloy, 
1990).  A  meta-analysis  by  Tett,  Jackson,  and  Rothstein  (1991)  provided  empirical 
evidence  that  personality  measures  can  be  used  to  predict  job  performance. 
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Most  of  the  meaningful  research  in  this  area  has  centered  on  the  five-factor  model 
of  personality:  neuroticism  (negative  emotionality  in  The  Big  Five),  extraversion, 
openness,  agreeableness,  and  conscientiousness  (Piedmont  and  Weinstein,  1994).  There 
are  two  models  that  have  found  the  most  favor  in  this  research.  Both  The  Big  Five 
model,  developed  by  Barrick  and  Mount  (1991),  and  the  NEO  PI-R  developed  by  Costa 
and  McCrae  (1992),  have  been  used  to  successfully  predict  job  performance.  Costa  and 
McCrae  have  also  established  that  the  five  dimensions  are  sufficient  to  describe  the 
information  contained  in  other  personality  systems  (1989).  A  third  model  which  closely 
mirrors  both  the  Big  Five  and  the  NEO  PI-R,  is  the  International  Personality  Inventory 
Protocol  (IPIP)  (Goldberg,  1999).  This  is  the  model  that  will  be  used  in  this  research. 

Some  researchers  feel  that  the  five-factor  model  is  inappropriate  for  use  as  a  tool 
to  predict  other  variables.  Block  (1995)  wrote  an  article  to  specifically  challenge  the 
usefulness  of  the  five-factor  models,  noting  that  five-factor  models  were  “not  specific 
enough”  to  represent  the  rich  realm  of  personality.  Hough  (1992)  felt  that  five-factor 
models,  in  particular  the  Big  5  (Barrick  and  Mount,  1991),  were  also  not  specific  enough. 
In  1995,  Schneider  and  Hough  noted  that  five-factor  models  were  too  expansive  to  link  to 
specific  behaviors. 

Rating  a  subordinate  is  the  job  of  raters  and  therefore  there  is  some  level  of  job 
performance  to  be  measured  as  they  make  their  rating  decisions.  Because  personality 
assessments  have  proven  useful  in  predicting  job  performance,  they  could  also  be  used  to 
predict  rater’s  assessments  of  subordinates.  A  recent  study  utilized  the  NEO  PI-R  to 
predict  supervisor  ratings  of  job  performance  based  on  a  personality  assessment  of  the 
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subordinate  (Piedmont  and  Weinstein,  1 994).  They  examined  the  correlations  of  the 
inventory  with  three  dimensions  of  job  performance  and  the  overall  rating.  They  found 
several  statistically  significant  correlations  between  the  personality  of  the  subordinate  and 
his  or  her  performance  ratings. 

The  next  step  in  this  line  of  research  would  be  to  see  if  the  personality  of  the 
raters  could  be  used  to  predict  their  ratings  or  their  rater  priorities.  Bernardin,  Cooke,  and 
Villanova  (2000)  recently  used  the  60-item  NEO  FFI  (Costa  and  McCrae,  1992)  to 
predict  rater  leniency.  They  examined  peer  ratings  and  professor  ratings  of  students  and 
found  both  to  be  significant.  They  found  that  the  personality  factors  Agreeableness  and 
Conscientiousness  were  significantly  correlated  to  overall  ratings  at  the  .05  level  (two 
tailed).  The  correlations  were  .33  and  -.37  respectively.  This  means  that,  relative  to  each 
other,  an  Agreeable  rater  would  tend  to  inflate  ratings  while  a  Conscientious  rater  would 
tend  to  deflate  ratings.  The  60-item  NEO  FFI  cannot  distinguish  between  the  individual 
facets  of  each  personality  factor,  however,  so  the  results  cannot  be  generalized  any 
further  than  the  overall  personality  factor. 

Mood  is  one  important  trait  that  may  affect  raters’  policy,  so  this  study  will 
capitalize  on  previous  research  that  provides  valid  measures  of  two  dimensions  of  mood, 
positive  and  negative  affect  (Watson,  Clark,  and  Tellegen,  1988).  “Positive  Affect 
reflects  the  extent  to  which  a  person  feels  enthusiastic,  active,  and  alert.  High  PA  is  a 
state  of  high  energy,  full  concentration,  and  pleasurable  engagement”  (Watson,  Clark, 
and  Tellegen,  1988:  1063).  “Negative  Affect  is  a  general  dimension  of  subjective 
distress  and  unpleasurable  engagement  that  subsumes  a  variety  of  aversive  mood  states, 
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including  anger,  contempt,  disgust,  guilt,  fear,  and  nervousness”  (Watson,  Clark,  and 
Tellegen,  1988:  1063). 

This  researcher  did  not  find  any  relevant  research  that  had  used  mood  in 
relationship  to  policy  capturing  or  performance  appraisals  from  the  rater  point  of  view. 
However,  based  on  the  definitions  of  positive  and  negative  affectivity,  these  measures 
might  have  a  relationship  with  rater  policy,  in  particular  rater  leniency  and  consistency. 
For  example,  a  high  PA  score  may  be  related  with  a  tendency  to  inflate  ratings  because 
the  respondent  focuses  on  the  more  positive  aspects  of  a  profile  while  a  high  NA  score 
may  show  the  opposite  tendency. 

The  mood  measures  will  be  measured  within  the  context  of  the  previous  year  of 
the  respondent.  This  is  an  effort  to  capture  the  long-term  mood  trait  of  the  respondent 
versus  a  short-term  mood.  By  expanding  the  time  frame  of  the  measures,  the  responses 
should  be  a  more  stable  representation  of  the  mood  traits  of  the  respondents  (Watson, 
Clark,  and  Tellegen,  1988;  Diener  &  Larsen,  1984;  and  Epstein,  1979) 

The  Four  Dimensional  Overall  Performance  Model  and  Associated  Behaviors 

Previous  research  provides  validation  to  the  use  of  a  four-dimensional  model  of 
overall  performance:  leadership,  task  performance,  interpersonal  facilitation,  and  job 
dedication.  Leadership  is  defined  as  “the  art  of  influencing  and  directing  people  to 
accomplish  the  mission”  (Air  Command  and  Staff  College,  1988:  1).  Task  performance 
is  the  proficiency  with  which  an  individual  performs  activities  that  are  formally 
recognized  as  the  technical  or  specialized  activities  that  define  his  or  her  job  (Borman  and 
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Motowidlo,  1993:  73).  Interpersonal  facilitation  is  the  extent  to  which  a  worker  supports 
other  members  of  the  organization  through  expressions  of  concern,  consideration, 
cooperative  and  helpful  acts  (Van  Scotter,  1994).  Job  dedication  is  the  extent  to  which  a 
worker  goes  above  and  beyond  the  requirements  of  his  or  her  official  job  description  and 
exceeds  the  minimum  expectations  of  his  or  her  rater.  Interpersonal  facilitation  and  job 
dedication  are  both  considered  contextual  behaviors.  The  individual  dimensions  are 
discussed  in  greater  detail  below. 

Contextual  behaviors  can  be  divided  into  two  categories:  job  dedication 
(behaviors  relating  to  the  organization)  and  interpersonal  facilitation  (behaviors  relating 
to  individuals)  (Motowidlo  and  Van  Scotter,  1994).  Katz  and  Kahn  (1978)  innovated  the 
field  of  behavior  by  differentiating  task  performance  from  “innovative  and  spontaneous 
behavior.”  The  spontaneous  behavior  includes  “cooperative  gestures,  actions  protecting 
the  organization,  and  behavior  that  enhances  the  external  image  of  the  organizations” 
(Katz  and  Kahn,  1978:  75-76).  Research  by  Katz  and  Kahn  (1978),  Smith  et  al.  (1983), 
Brief  and  Motowidlo  (1986),  Motowidlo  et  al.  (1986),  Day  and  Silverman  (1989), 
Campbell  and  Zook  (1991),  Borman  and  Motowidlo  (1993),  and  Van  Scotter  (1994) 
provide  empirical  evidence  that  combining  interpersonal  facilitation  behaviors  and  job 
dedication  behaviors  into  a  single  measure  of  contextual  performance  may  obscure  any 
assessment  of  individual  effectiveness.  Van  Scotter  determined  that  task  performance, 
interpersonal  facilitation,  and  job  dedication  are  all  separate  dimensions  of  individual 
effectiveness  and  that  all  individually  contribute  to  the  organizational  effectiveness  of  Air 
Force  maintenance  technicians  (1994).  Research  by  Hurry  indicated  that  a  fourth  factor, 
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leadership,  could  be  added  as  a  separate  dimension  of  individual  effectiveness  (1995). 
Hurry  found  that  the  intercorrelation  between  leadership  and  the  other  three  dimensions 
of  individual  effectiveness  was  small,  indicating  that  leadership  is  an  orthogonal 
dimension  of  overall  performance.  In  other  words,  leadership  can  be  treated  as  a 
separate,  distinct  dimension  of  overall  performance.  Because  Hurry  (1995)  depended  on 
the  four-dimensional  model,  and  this  research  is  attempting  to  validate  her  efforts,  it  is 
necessary  for  this  research  to  also  depend  on  the  four-dimensional  model  of  overall 
performance. 

Not  all  research  agrees  with  the  four-factor  model  of  performance,  though. 
Podsakoff,  Mackenzie,  Paine,  and  Bachrach  (2000)  noted  that  job  dedication  and 
interpersonal  facilitation  might  not  contribute  to  overall  performance  independently 
enough  to  be  measured  separately  because  it  “is  difficult  to  distinguish  empirically  from 
in-role  or  task  performance”.  James  Conway  (1999)  tested  whether  job  dedication  and 
interpersonal  facilitation  contributed  independently  to  overall  performance,  hypothesizing 
that  they  would  be  redundant.  He  found  that  there  is  some  evidence  to  suggest  that  the 
two  dimensions  do  contribute  independently  to  overall  performance  and  that  supervisors 
pay  more  attention  to  task  performance. 

There  is  a  plethora  of  research  concerning  behaviors  that  are  indicative  of  the  four 
dimensions  of  overall  performance  examined  in  this  study  (Bausum,  1986;  Borman  and 
Motowidlo,  1993;  Air  Command  and  Staff  College,  1988;  Conger  et  ah,  1989;  Van 
Scotter  and  Shane,  1995;  Borman  and  Brush,  1993;  Department  of  the  Air  Force,  1988; 
and  Van  Scotter,  1994).  The  behaviors  were  revalidated  when  examined  by  Hurry 
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(1995).  The  behaviors  chosen  by  Hurry  (1995)  for  leadership,  task  performance, 
interpersonal  facilitation,  and  job  dedication  and  the  behavior  means  as  found  by  Hurry 
are  listed  in  Appendix  A  through  Appendix  D  respectively.  Following  is  a  more  detailed 
discussion  concerning  each  of  the  dimensions. 

Leadership. 

Hogan  et  al.  proposed  that  leadership  “involves  persuading  other  people  to  set 
aside  for  a  period  of  time  their  individual  concerns  and  to  pursue  a  common  goal” 

(Hogan  et  al.,  1994:  493).  The  Air  Force  refines  the  definition  of  leadership  to  “the  art  of 
influencing  and  directing  people  to  accomplish  the  mission”  (Air  Command  and  Staff 
College,  1988:  1).  Bausum  (1986),  Borman  and  Motowidlo  (1993),  Air  Command  and 
Staff  College  (1988),  Conger  et  al.  (1989),  Borman  and  Brush  (1993)  and  Van  Scotter 
and  Shane  (1995)  all  developed  lists  of  effective  leader  behaviors.  Subsequent  research 
pooled  these  lists  and  developed  a  common  list  of  the  behaviors  most  relevant  to  junior 
officers  in  the  Air  Force  (Hurry,  1995). 

Task  Performance. 

Borman  and  Motowidlo  define  task  performance  as  the  proficiency  with  which  an 
individual  performs  activities  that  are  formally  recognized  as  the  technical  or  specialized 
activities  that  define  his  or  her  job  (Borman  and  Motowidlo,  1993:  73).  Task 
performance  behaviors  contribute  directly  and  indirectly  to  the  organization’s  technical 
core  processes  and  to  the  production  of  goods  or  services  through  proficient  and  effective 
behaviors  (Van  Scotter,  1994:  23). 
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Campbell  et  al.  assert  that  proficiency  is  an  important  component  of  task 
performance  behaviors  and  states  that  core  technical  proficiency  “refers  to  how  well  the 
individual  can  execute  the  core  technical  task  the  job  requires,  given  a  willingness  to  do 
so”  (Campbell  et  al.,  1990:  322).  Lists  of  task  performance  behaviors  have  been 
developed  by  the  Department  of  the  Air  Force  (1988),  Van  Scotter  (1994)  and  Borman 
and  Brush  (1993).  These  behaviors  were  compiled  and  distilled  into  a  list  by  Hurry 
(1995).  These  behaviors  do  not  indicate  specific  job  skills,  but  state  how  well  the 
individual  performs  his  or  her  core  job  tasks  in  general. 

Interpersonal  Facilitation. 

Van  Scotter  defined  interpersonal  facilitation  as  the  extent  to  which  a  worker 
supports  other  members  of  the  organization  through  expressions  of  concern, 
consideration,  cooperative  and  helpful  acts.  The  individual  also  contributes  to  co- 
workers’  effective  task  performance,  encourages  others  to  perform  in  organizationally 
relevant  ways,  and  helps  to  maintain  a  social  and  psychological  climate  that  facilitates 
accomplishment  of  the  organization’s  goals  (Van  Scotter,  1994:  21-24).  Interpersonal 
facilitation  is  behaviors  that  increase  the  organizations  overall  performance,  but 
individuals  receive  little,  if  any,  direct  compensation  for  the  performance  of  these 
behaviors.  Positive  interpersonal  facilitation,  though,  is  one  facet  of  an  overall  effective 
performer. 

Organ  introduced  the  characteristics  of  organizational  citizenship  behaviors  that 
are  indicative  of  interpersonal  facilitation  including  altruism,  courtesy,  and  sportsmanship 


20 


(Organ,  1990:  96).  For  the  purposes  of  this  study,  altruism  is  defined  as  “voluntary 
actions  that  help  another  person  with  a  work  problem,”  while  courtesy  “subsumes  all  of 
the  foresightful  gestures  that  help  someone  else  prevent  a  problem”  (Organ,  1990:  96). 

He  defined  sportsmanship  as  “a  citizen-like  posture  of  tolerating  the  inevitable 
inconveniences  and  impositions  of  work  without  whining  and  grievances”  (Organ,  1990: 
96). 

Borman  and  Motowidlo  (1993),  Van  Scotter  (1994),  Van  Scotter  and  Shane 
(1995),  and  Borman  and  Brush  (1993)  each  described  a  set  of  behaviors  associated  with 
effective  interpersonal  facilitation.  These  behaviors  were  compiled  and  distilled  by 
Hurry  (1995). 

Job  Dedication. 

Job  dedication  “transcends  job  involvement  and  motivations  to  perform  the 
specific  tasks  that  comprise  the  job  and  connotes  a  sense  of  loyalty  to  the  organization  as 
a  whole  and  a  desire  to  fulfill  more  general  role  requirements  that  come  with 
organizational  membership”  (Borman  and  Motowidlo,  1993:  78).  Hurry  acknowledged 
that  job  dedication  “encompasses  volitional,  motivated  behaviors  that  are  driven  by  will, 
motivational  orientations,  and  beliefs  about  the  value  of  work”  (Hurry,  1995:  20).  Self- 
motivation,  working  hard,  attending  to  important  details,  and  persisting  to  finish  a 
difficult  task  contribute  to  individual  and  organizational  effectiveness  (Van  Scotter,  1994: 
2).  Most  discussions  of  job  dedication  revolve  around  describing  behavior  that  is 
indicative  of  the  dimension.  Therefore,  this  researcher  has  defined  job  dedication  as  the 
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extent  to  which  a  worker  goes  above  and  beyond  the  requirements  of  his  or  her  official 
job  description  and  exceeds  the  minimum  expectations  of  his  or  her  rater. 

Borman  and  Motowidlo  (1993),  Van  Scotter  (1994),  Van  Scotter  and  Shane 
(1995),  and  Borman  and  Brush  (1993)  each  developed  behaviors  that  are  indicative  of  job 
dedication.  These  behaviors  were  assembled  and  refined  by  Hurry  (1995). 

The  Foundation  of  This  Research 

This  research  is  primarily  built  upon  the  foundation  of  research  by  Hurry  (1995) 
and  hopes  to  improve  that  research  by  eliminating  the  order  effect  present  in  her  survey 
instrument  and  examining  personality  traits  as  possible  predictors  of  rating  policy.  The 
four-factor  model  proposed  by  Hurry  (1995)  and  utilized  by  this  thesis  is  shown  in  Figure 
2  (Hurry,  1995). 


Figure  2.  Four-Factor  Model  of  Junior  Officer  Performance  (Hurry,  1995) 
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Hurry  developed  a  survey  with  50  hypothetical  officer  profiles.  Each  profile 
listed  one  example  from  each  of  the  four  behaviors  and  asked  respondents  to  rate  the 
behavior  and  provide  an  overall  performance  rating  for  the  hypothetical  officer. 

The  [Pid,  Ptp,  Pif,  and  Pjd]  in  this  equation  represent 
the  relative  weights  (or  amount  of  importance)  the 
supervisors  place  on  a  given  performance  category. 

Therefore,  this  model  says  that  an  individual’s  overall 
effectiveness  is  the  sum  of  his  or  her  performance  in 
leadership,  task  performance,  job  dedication,  and 
interpersonal  facilitation  multiplied  by  the  weight  their 
supervisor  places  on  the  particular  category.  The  average 
weights  for  each  category  over  a  large  sample  of 
supervisors  represent  the  organization’s  policy.  However, 
if  supervisor’s  views  of  performance  vary  greatly  or  the 
performance  factors  overlap  too  much,  one  or  more  of  the 
factors  may  not  be  significantly  different  from  zero.  Thus, 
the  model  provides  a  framework  for  determining  which 
classes  of  behavior  Air  Force  supervisors  judge  as  most 
important  for  effective  junior  officer  performance.  (Hurry, 

1995:23) 

She  found  that  the  order  of  importance  that  raters  placed  on  the  four  individual 
behaviors  was  leadership,  task  performance,  interpersonal  facilitation,  and  job  dedication 
respectively.  However,  instrument  effects  may  have  confounded  her  method.  The 
survey  used  for  Hurry’s  (1995)  study  listed  the  four  behaviors  in  the  same  order  for  every 
hypothetical  officer  profile.  This  order  was  identical  to  the  order  of  importance  that  was 
found  in  the  results;  therefore  the  results  cannot  be  relied  upon  until  validated. 


Objectives  and  Hypothesis 

This  study’s  objectives  are:  1)  to  determine  the  knowledge,  skills,  and  behaviors 
that  supervisors  feel  contribute  the  most  to  the  performance  of  junior  officers,  2)  to 
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explore  the  possibility  that  rater  characteristics  (i.e.  experience,  training,  demographic 
variables,  etc.)  may  influence  appraisal  policy,  3)  to  determine  if  officer  career  field  has 
any  effect  on  appraisal  decisions,  4)  to  test  the  relationship  between  rater  personality 
traits  and  ratings  given  to  ratees,  and  5)  to  determine  if  ratee  gender  has  any  influence 
over  the  individual  dimension  or  overall  performance  ratings.  To  accomplish  these 
objectives,  the  following  research  questions  and  accompanying  hypotheses  will  be 
investigated.  It  is  proper  to  recognize  that  questions  1-5  and  hypotheses  1-9  are 
significantly  the  same  as  the  questions  and  hypotheses  posed  by  Hurry  (1995). 

Question  1 .  Which  leadership,  task  performance,  interpersonal  facilitation,  and 
job  dedication  behaviors  are  most  important  in  accomplishing  Air  Force  jobs? 

Question  2.  Do  leadership,  task  performance,  interpersonal  facilitation,  and  job 
dedication  behaviors  each  contribute  significantly  and  independently  to  supervisor’s 
overall  performance  rating  decisions? 

Hypothesis  1 :  Supervisors  will  use  all  four 
performance  dimensions  while  making  decisions  about 
overall  junior  officer  performance.  The  mean  standardized 
beta  weights  for  leadership,  task  performance,  interpersonal 
facilitation,  and  job  dedication  will  all  be  significantly 
different  from  zero  when  the  dimension  ratings  are 
regressed  against  the  overall  performance  ratings. 

Question  3.  Does  the  importance  of  the  four  dimensions  of  performance — 
leadership,  task  performance,  interpersonal  facilitation,  and  job  dedication — change 
depending  on  officer  occupation? 

Hypothesis  2:  The  importance  of  the  four 
performance  dimensions  will  significantly  differ  by  officer 
occupation. 
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Question  4.  Does  the  importance  of  the  four  dimensions  or  the  mean  score  of  the 
four  dimensions  or  overall  score  vary  with  the  supervisor’s  grade,  race,  sex,  or 
commissioning  source? 

Hypothesis  3 :  The  beta  weights  and  the  mean  scores 
for  the  four  dimensions  of  performance  and  the  mean 
scores  for  the  overall  performance  will  vary  with  officer 
experience/grade. 

Hypothesis  4:  The  beta  weights  for  the  four 
dimensions  of  performance  will  vary  with  the  supervisor’s 
sex. 


Hypothesis  5 :  The  beta  weights  and  mean  scores  for 
the  four  dimensions  of  performance  and  the  mean  scores 
for  the  overall  performance  will  vary  with  the  supervisor’s 
commissioning  source. 

Question  5.  Does  the  consistency  of  the  captured  rating  policy  vary  with  the 
supervisor’s  job  category,  grade,  race,  sex,  experience,  or  commissioning  source? 

Hypothesis  6:  The  consistency  of  the  rating  policy 
will  differ  between  job  categories. 

Hypothesis  7:  The  consistency  of  the  rating  policy 
will  vary  with  the  experience/grade  of  the  raters. 

Hypothesis  8:  The  consistency  of  the  rating  policy 
will  differ  between  genders  of  the  raters. 

Hypothesis  9:  The  consistency  of  the  rating  policy 
will  differ  between  commissioning  sources  of  the  raters. 

Question  6.  Can  the  personality  characteristics  of  the  rater  be  used  to  accurately 
predict  some  variance  in  the  ratings  given  for  the  four  performance  dimensions  or  the 
overall  performance  ratings? 

Hypothesis  10:  Some  or  all  of  the  personality 
characteristics  of  the  rater  will  be  statistically  significant 


25 


predictors  of  dimension  ratings  and  overall  performance 
ratings. 

Questions  7.  Does  the  gender  of  the  ratee  affect  the  ratings  received  in  any  of  the 
four  performance  dimensions  or  the  overall  performance  rating? 

Hypothesis  1 1 :  The  gender  of  the  ratee  will  not 
affect  the  ratings  received  in  the  four  performance 
dimensions  or  the  overall  performance  ratings. 
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III.  Method 


The  Survey  Instrument 

The  first  step  in  this  policy  capturing  study  was  to  gather  behaviors  that  were 
indicative  of  junior  officer  behaviors.  These  behaviors  have  been  previously  classified 
into  the  four  dimensions  of  overall  performance  and  validated  by  a  number  of  researchers 
(Hurry,  1995;  Bausum,  1986;  Borman  and  Motowidlo,  1983;  Air  Command  and  Staff 
College,  1988;  Conger  et  al.,  1989,  Van  Scotter  and  Shane,  1995;  Borman  and  Brush, 
1993;  Van  Scotter,  1994;  and  Department  of  the  Air  Force,  1988).  In  1995,  Hurry 
performed  a  survey  of  84  Air  Force  officers  where  the  officers  were  tasked  to  rate  the 
importance  of  1 00  behaviors  selected  from  these  validated  behaviors  because  of  their 
particular  relevance  to  Air  Force  junior  officers.  The  behaviors  indicated  four 
dimensions  of  overall  performance.  These  behaviors  and  their  mean  importance  as  found 
in  this  survey  are  listed  in  Appendix  A  through  Appendix  D. 

Pilot  Survey 

The  pilot  survey  for  this  research  is  not  provided  for  reference  because  it  is  very 
similar  to  the  full  survey  included  in  Appendix  F.  The  pilot  survey  was  developed  to  test 
the  survey  instrument,  ask  for  suggestions  on  the  survey  instrument,  check  for  the 
necessary  sample  size  for  the  full  survey  instrument,  and  provide  some  preliminary 
results  to  test  the  analysis  tools  chosen  for  this  research.  Two  versions  of  the  survey  were 
created.  The  first  version  will  be  referred  to  as  the  normal  version  and  the  second  version 
will  be  referred  to  as  the  reverse-ordered  version.  The  second  version  is  reverse-ordered 
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and  the  12  female  profiles  on  this  survey  correspond  to  12  identical  male  profiles  on  the 
normal  version.  The  12  female  profiles  on  the  normal  version  likewise  correspond  to  12 
identical  male  profiles  on  the  reverse-ordered  version. 

Sample  and  Procedure. 

Twenty-eight  Air  Force  officers  in  the  ranks  of  Captain  through  Colonel  were 
selected  randomly  from  the  population  of  faculty  and  staff  officers  at  the  Air  Force 
Institute  of  Technology,  making  this  a  sample  of  convenience.  These  officers  were 
informed  that  completion  of  the  survey  was  voluntary.  A  total  of  1 7  officers  completed 
and  returned  the  survey  for  a  61%  response  rate.  The  surveys  were  placed  in  each 
officer’s  mailbox  and  returned  to  the  researcher  through  the  internal  mail  distribution 
system.  Half  (14)  of  the  officers  received  the  normal  survey  and  the  other  half  (14) 
received  the  reverse-ordered  survey. 

Survey  Instrument. 

The  first  part  of  the  survey  was  devoted  to  gathering  the  demographic  information 
necessary  to  test  for  group  differences  in  rating  policy. 

The  profiles  were  developed  through  a  series  of  steps.  The  first  step  was  to 
design  a  scenario  that  could  realistically  be  encountered  by  an  Air  Force  officer.  This 
was  attempted  because  of  the  emphasis  placed  on  realism  by  Hobson  and  Gibson  (1983). 
The  scenario  in  Figure  4  at  the  top  of  the  next  page  was  developed  for  this  purpose. 
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You  have  just  been  selected  to  validate  ratings  given  to  junior  officers  that  attended  Squadron 
Officer  School  (SOS).  For  some  time,  SOS  has  been  concerned  about  the  validity  of  the 
performance  ratings  that  SOS  flight  commanders  provide  for  their  flight  members.  To  validate  past 
performance  ratings,  SOS  has  requested  your  help. 

SOS  has  provided  you  with  a  list  of  profiles.  Each  of  these  profiles  has  four  behaviors  that  were 
extracted  from  actual  training  reports.  All  of  the  behaviors  fit  into  four  broad  categories  of 
behaviors  as  follows: 

Leadership — The  art  of  influencing  and  directing  people  to  accomplish  the  mission. 

Task  Performance — The  proficiency  with  which  an  individual  performs  activities  that  are  formally 
recognized  as  the  technical  or  specialized  activities  that  define  his  or  her  job. 

Interpersonal  Facilitation — The  extent  to  which  a  worker  supports  other  members  of  the 
organization  through  expressions  of  concern,  consideration,  cooperative  and  helpful  acts. 

Job  Dedication — The  extent  to  which  a  worker  goes  above  and  beyond  the  requirements  of  their 
official  job  description  and  exceeds  the  minimum  expectations  of  their  rater. 

Keep  in  mind  that  all  of  the  behaviors  you  will  see  in  the  following  profiles  are  positive  behaviors 
(in  keeping  with  Air  Force  training  report  and  OPR  tradition).  Some  of  the  behaviors,  though,  are 
“more  positive”  than  others  and  so  should  be  more  highly  rated  relative  to  those  “less  positive” 
behaviors. 

The  category  of  each  behavior  is  also  provided  with  each  profile. 

Your  job  is  simple.  Read  each  behavior  and  then  rate  the  behavior  using  the  rating  scale  provided. 
The  scale  is  a  5-point  scale  with  the  value  of  the  behavior  increasing  as  you  move  to  the  right.  A 
rating  in  the  far  right  box  would  indicate  above  average  performance,  while  a  rating  in  the  far  left 
box  would  indicate  average  performance.  Place  an  X  in  the  box  that  you  feel  best  describes  the 
behavior  or  overall  rating.  Then  provide  an  overall  rating  for  the  profile  based  upon  your  evaluation 
of  the  four  behaviors.  There  are  48  profiles. 


Figure  3.  Survey  Scenario 


The  second  step  was  to  group  the  behaviors  together  into  profiles  so  that  each 
profile  was  represented  by  one  behavior  from  each  of  the  four  dimensions  of 
performance.  The  behaviors  gathered  and  rated  for  importance  by  Hurry  were  divided 
into  quartiles  based  upon  their  mean  importance  (1995).  Then,  the  behaviors  were 
systematically  assigned  to  profiles  so  that  each  profile  would  be  average.  This  was 
accomplished  by  combining  the  behaviors  so  that  each  profile  had  a  behavior  from  a 
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separate  quartile  of  each  dimension.  For  example,  profile  one  would  have  a  leadership 
behavior  from  the  fourth  quartile,  a  task  performance  behavior  from  the  third  quartile,  an 
interpersonal  facilitation  behavior  from  the  second  quartile,  and  finally  a  job  dedication 
from  the  first  quartile.  This  resulted  in  24  unique  combinations  of  behaviors  from  the 
quartiles  and  these  same  combinations  were  repeated  for  24  more  profiles  resulting  in  48 
total  profiles.  Table  1  shows  an  example  of  the  combinations  of  behaviors  used  to 
produce  the  first  six  profiles. 


Table  1.  Behavior  Consolidations  for  Profile  Construction 


Behavior  Dimension 

Profile  1 

Profile  2 

Profile  3 

Profile  4 

Profile  5 

Profile  6 

Leadership  Quartile 

4 

4 

4 

4 

4 

4 

Task  Performance  Quartile 

3 

3 

2 

2 

1 

1 

Interpersonal  Facilitation 
Quartile 

2 

1 

1 

3 

2 

3 

Job  Dedication  Quartile 

1 

2 

3 

1 

3 

2 

These  combinations  of  behaviors  resulted  in  profiles  that  represent  an  average 
junior  officer.  This  should  ensure  that  any  differences  found  in  importance  should  be 
attributable  to  the  importance  of  the  behaviors,  and  not  the  spurious  result  of  a  different 
type  of  combination  where  one  profile  has  four  very  important  behaviors  or  four 
unimportant  behaviors. 

After  combining  the  behaviors  into  profiles,  the  behaviors  were  randomly 
shuffled  so  that  no  distinct  order  of  behavior  dimensions  would  be  evident.  This  step 
overcomes  the  weakness  found  in  a  previous  study  where  the  same  order  of  behaviors 
was  used  for  each  profile.  The  resulting  order  of  importance  of  the  dimensions  mirrored 
the  order  of  the  behaviors  within  the  profiles  (Hurry,  1995). 
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The  next  step  entailed  assigning  last  names  to  the  profiles.  A  list  of  last  names 
was  generated  alphabetically  and  a  last  name  was  assigned  to  each  profile.  Then  the 
profiles  were  randomly  shuffled  to  remove  that  alphabetical  order  of  the  last  names. 

Finally,  a  list  of  96  first  names  was  generated  for  the  gender  portion  of  this  study. 
Of  the  96,  half  were  male  names  and  half  were  female  names.  These  names  were 
incorporated  into  a  questionnaire  and  30  respondents  (chosen  for  convenience)  were 
asked  to  rate  whether  a  name  was  male,  female,  or  could  easily  be  male  or  female.  Only 
those  names  with  1 00%  inter-rater  agreement  on  gender  were  chosen  for  use  in  this 
study.  These  first  names  were  assigned  to  a  duplicate  survey  using  36  male  names  and 
12  female  names.  Twelve  female  names  were  chosen  for  each  survey  because  the  ratio  of 
female  names  to  male  names  approximates  the  ratio  of  females  to  males  in  the  Air  Force. 

The  next  portion  of  the  survey  instrument  was  devoted  to  measuring  several 
personality  trait  facets.  This  researcher  chose  facets  from  the  International  Personality 
Item  Pool  (IPIP).  It  was  necessary  to  limit  the  facets  in  an  effort  to  keep  the  survey  to  a 
reasonable  length,  so  facets  were  chosen  for  their  interest  to  the  research  at  hand.  The 
facets  chosen  included  Friendliness,  Assertiveness,  Intellect,  Trust,  Altruism,  Self- 
Efficacy,  and  Cautiousness.  Each  of  these  facets  includes  ten  measurement  items.  The 
70  personality  measures  were  randomized  and  scored  on  a  5  point  Likert  scale. 

The  final  portion  of  the  personality  assessment  was  the  Brief  Measures  of  Positive 
and  Negative  Affect:  The  PANAS  Scales  (Watson,  Clark,  and  Tellegen,  1988).  These 
items  were  measured  using  the  time  frame  “the  last  year”.  This  way,  the  measures  better 
reflect  long-term  traits  versus  short-term  moods. 
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The  last  section  of  the  pilot  survey  was  a  section  requesting  respondents  write 
their  feelings  about  the  survey:  likes,  dislikes,  and  areas  that  may  need  improvement. 
These  suggestions  were  used  to  refine  the  full  survey  as  discussed  later. 

A  second  survey  was  created  with  the  profiles  reverse-ordered  to  remove  any 
order  effects.  The  profiles  were  also  assigned  36  male  names  and  12  female  names. 
However,  this  survey  was  designed  so  that  each  female  profile  corresponded  to  a  male 
profile  on  the  other  survey.  This  allows  a  paired  sample  analysis  to  be  performed  to 
check  for  ratee  gender  effects. 

Final  Survey 

The  final  survey  is  similar  to  the  pilot  survey,  although  some  suggestions  were 
incorporated  from  the  pilot  and  will  be  discussed  next.  The  final  survey  instrument  is 
included  for  reference  in  Appendix  F. 

Suggestions  and/or  Comments  Received  on  the  Pilot  Survey. 

The  following  suggestions  were  received  from  respondents  of  the  pilot  survey. 
The  response  (R)  to  each  comment  (C)  immediately  follows  the  comment. 

(C)  Did  participants  complete  the  survey  in  same  environment? 

(R)  Although  a  valid  concern,  this  cannot  be  controlled  for  on  a  mail  survey. 
(C)  How  many  participants  put  the  survey  down  and  came  back  to  it? 

(R)  Although  a  valid  concern,  this  cannot  be  controlled  for  on  a  mail  survey. 
(C)  Time,  time,  time! 
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(R)  This  survey  was  long,  but  that  is  a  product  of  the  necessity  to  have  1 0  items 
for  each  variable  being  measured. 

(C)  Why  not  use  1-3  scale  versus  1-5  scale  on  the  [personality  measures]? 

(R)  These  items  were  measured  as  suggested  by  the  researchers  who  developed 
these  scales,  using  the  directions  provided  by  them. 

(C)  Will  there  be  low  variability  because  of  no  negative  behaviors  in  profiles? 

(R)  This  is  a  valid  comment  and  is  a  weakness  of  this  study.  This  is  discussed  in 
further  detail  in  Chapter  V.  However,  one  point  of  this  research  is  to  validate  or 
invalidate  a  previous  research  effort  that  used  only  positive  behaviors.  If  this 
research  varies  too  far  from  that  original  research  effort,  it  cannot  be  reliably 
compared  to  it. 

(C)  Have  a  practice  page. 

(R)  The  survey  is  already  too  long  and  a  practice  page  would  make  it  even  longer. 
(C)  Repeated  behaviors  seemed  excessive. 

(R)  This  comment  led  to  a  reevaluation  of  the  profile  construction  and  an  error 
was  discovered  in  the  way  they  were  created.  This  will  be  discussed  later  in  this 
chapter  under  Changes  Made  for  the  Final  Survey. 

(C)  The  survey  does  not  capture  the  added  assessment  consideration  when  a 
junior  officer  is  filling  a  certain  type  of  job  that  may  require  added  excellence  in 
one  or  more  dimensions  by  nature  of  the  job  itself. 
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(R)  This  is  an  insightful  comment  on  one  of  the  weaknesses  of  this  survey,  its 
generalizability  to  the  real  world  of  performance  appraisals.  This  weakness  will 
be  discussed  in  Chapter  V,  Discussion  and  Conclusions. 

Changes  Made  for  the  Final  Survey. 

The  most  notable  change  from  the  pilot  survey  to  the  final  survey  was  the  change 
in  the  profile  construction.  One  of  the  comments  received  [multiple  times]  on  the  pilot 
survey  led  this  researcher  to  examine  the  profile  construction.  An  error  was  made  in  the 
construction  resulting  in  much  more  duplication  of  behaviors  and  profiles  than  was 
intended.  This  error,  combined  with  the  method  of  selecting  combinations  of  behavior 
resulted  in  an  unacceptably  high  inter-item  correlation  between  profiles,  correlations 
greater  than  .3.  This  would  result  in  a  high  level  of  interdependence  between  dimensions 
in  the  results  and  mask  the  true  contributions  of  each  dimension  to  the  overall 
performance  rating.  Two  steps  were  taken  to  alleviate  this  problem.  First,  the  profiles 
were  reconstructed  to  remove  duplicate  profiles  and  ensure  that  no  individual  behavior 
was  repeated  more  than  twice.  Only  12  of  the  92  behaviors  were  repeated  twice,  and  the 
others  were  only  repeated  once.  This  resulted  in  an  inter-item  correlation  that  was  still 
too  high  (see  Table  2).  As  you  can  see,  most  of  the  correlation  problem  stems  from  the 
leadership  dimension.  Therefore,  the  leadership  behaviors  were  randomly  shuffled  and 
reassigned  to  the  profiles.  This  resulted  in  an  acceptable  inter-item  correlation  while 
maintaining  as  much  of  the  original  effort  to  ensure  “average”  profiles  as  possible.  Table 
3  shows  the  resulting  correlations. 
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Table  2.  Pilot  Between  Profile  Correlation  of  Behavior  Dimensions  (high 

correlation) 


Leadership 

Task 

Performance 

Interpersonal 

Facilitation 

Job 

Dedication 

Leadership 

1 

Task  Performance 

-0.163371 

1 

Interpersonal  Facilitation 

0.3129146 

0.037227 

1 

Job  Dedication 

-0.160955 

-0.000242 

0.0176697 

1 

n=48 

Table  3.  Final  Survey  Between  Profile  Correlation  of  Behavior  Dimensions 


Leadership 

Task 

Performance 

Interpersonal 

Facilitation 

Job 

Dedication 

Leadership 

1 

Task  Performance 

-0.11162 

1 

Interpersonal  Facilitation 

0.031456 

0.037227 

1 

Job  Dedication 

-0.09429 

-0.00024 

0.01767 

1 

3 

II 

OO 

Another  important  change  to  the  final  survey  was  the  decision  to  concentrate  on 
only  the  Conscientiousness  factor  of  the  IPIP  personality  measure.  Partially  measuring  4 
different  factors  resulted  in  an  incomplete  picture,  but  the  lengthiness  of  the  survey  made 
it  impractical  to  add  all  the  items  to  measure  all  4  factors.  Therefore,  the 
Conscientiousness  factor  was  chosen  to  be  measured  fully,  using  all  6  facets:  Self- 
Efficacy,  Orderliness,  Dutifulness,  Achievement  Striving,  Self-Discipline,  and 
Cautiousness.  This  resulted  in  60  items  randomized  into  the  full  survey.  The 
Conscientiousness  factor  was  chosen  because  it  showed  the  most  promise  for  significant 
results  in  the  pilot  survey  analysis  and  in  previous  research  on  rater  leniency  (Bernardin, 
Cooke,  and  Villanova,  2000). 
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The  other  changes  to  the  survey  were  made  for  aesthetic  and  user-friendliness 
reasons.  The  instructions  were  simplified  to  reduce  the  amount  of  reading  required  by 
the  respondents.  The  profiles  were  changed  to  present  more  delineation  between  the 
different  behaviors,  to  heighten  the  impact  of  the  name  associated  with  each  profile,  and 
to  make  each  profile  separate  and  more  visually  pleasing. 

Sample  and  Procedure. 

Wright-Patterson  AFB,  OH,  Maxwell  AFB,  AL,  Langley  AFB,  VA,  and  Peterson 
AFB,  CO  were  selected  for  this  survey  as  they  each  are  indicative  of  one  of  the  four 
following  major  commands:  Air  Force  Materiel  Command  (AFMC),  Air  Combat 
Command  (ACC),  Air  Force  Space  Command  (AFSPC),  and  Air  Education  and  Training 
Command  (AETC).  100  officers  from  each  base  were  randomly  selected  to  participate  in 
the  survey  from  the  population  of  officers  in  the  grades  of  Captain  (greater  than  7  years) 
through  Lieutenant  Colonel.  Half  of  the  officers  at  each  base  received  the  normal  survey 
instrument,  while  the  other  half  received  the  reverse-ordered  survey  instrument.  1 00 
students  from  AFIT  in  the  grades  of  Second  Lieutenant  through  Major  also  received  the 
survey.  Half  received  the  normal  version  and  half  received  the  reverse-ordered  version. 
Finally,  50  students  attending  Defense  Acquisition  University  continuing  education 
courses  received  the  survey;  half  received  the  normal  version  and  half  received  the 
reverse-ordered  version.  The  DAU  and  AFIT  students  were  a  sample  of  convenience 
intended  to  supplement  the  mail  survey  and  bolster  the  sample  size  in  the  event  of  a  poor 
response  rate. 
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One  hundred  and  forty-eight  of  the  400  surveys  were  returned  from  the  mail 
distribution  to  the  four  different  Air  Force  bases  for  a  37  percent  response  rate.  Of  the 
148,  12  were  deemed  unusable  because  they  were  incorrectly  filled  out.  Fifty-six  of  the 
100  surveys  were  returned  from  the  AFIT  students,  a  56  percent  response  rate.  Of  the  56 
surveys,  4  were  deemed  unusable  because  they  were  improperly  completed.  Twenty-four 
of  the  50  DAU  students  returned  the  survey,  a  48  percent  response  rate.  Of  the  24,  3 
were  deemed  unusable  due  to  missing  data. 

Analysis 

All  analyses  in  this  research  will  be  performed  using  SPSS,  Version  10.0  for 
Windows®,  Microsoft  Excel  2000,  Version  9.0.2720,  and  JMPIN,  Version  3.2.6.  This 
research  will  use  multiple  regression  analysis  to  test  whether  leadership,  task 
performance,  interpersonal  facilitation,  and  job  dedication  each  explain  a  unique  portion 
of  the  variance  in  dimension  ratings  and  overall  performance  ratings. 

The  first  step  in  the  analysis  was  the  preliminary  work  to  ensure  the  assumptions 
of  normality  and  homoscedasticity  are  met.  Then,  a  reliability  analysis  of  the  personality 
facets,  PA  and  NA,  was  performed  to  ensure  that  the  responses  display  reliability 
commensurate  with  published  work  on  these  items.  Then  the  personality  items  were 
combined  into  their  respective  facets.  The  facets  were  combined  into  their  factor,  and  the 
PA  and  NA  items  were  summated  into  their  scales.  This  test  for  reliability  was 
conducted  with  Cronbach’s  alpha  as  the  measurement  statistic. 
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The  profiles  for  each  returned  survey  were  analyzed  to  develop  a  regression 
equation  for  each  respondent.  This  equation  represented  the  rating  policy  for  that 
respondent.  The  unstandardized  beta  weights  for  each  dimension  (leadership,  task 
performance,  ptpi  interpersonal  facilitation,  (3jf;  and  job  dedication,  (3jd)  represented  the 
relative  importance  of  each  performance  dimension  to  the  dependent  variable,  the  overall 
performance  rating.  The  equation  included  a  term  for  each  behavior  dimension,  a  term 
for  the  order  of  each  dimension  in  the  profile,  a  term  for  the  gender  of  the  profile,  two 
terms  for  the  minimum  and  maximum  ratings  given,  and  a  full  interaction  examination 
between  gender  and  each  of  the  four  dimensions  of  behavior. 

The  beta  weights  for  each  respondent  were  combined  into  a  data  set  for  further 
regression  analysis.  First,  the  beta  weights  were  subjected  to  an  analysis  of  means  to 
identify  if  the  mean  importance  of  each  dimension  differed  from  the  others  at  the  sample 
level.  Then,  the  beta  weights  for  each  respondent  were  combined  with  their  demographic 
data,  personality  facet  scores,  and  PA  and  NA  scores.  The  personality  facets  were  also 
aggregately  examined  at  the  factor  level.  Then,  several  regressions  were  performed. 

The  first  was  with  adjusted  R  as  the  dependent  variable  (as  a  proxy  for  rating 
consistency)  to  test  whether  rank  or  any  other  variable  influenced  the  consistency  of  the 
rater.  A  regression  was  also  performed  to  see  if  any  of  the  variables  could  be  used  to 
predict  rating  leniency,  as  measured  by  average  overall  performance.  The  next  four 
regressions  included  each  behavior  beta  weight,  in  turn,  to  see  if  any  of  the  other 
variables  could  be  used  to  predict  the  beta  weights.  This  analysis  tested  the  same 
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questions  of  variance,  except  the  results  identified  if  any  other  measured  variables  could 
be  used  to  predict  beta  weights. 

Finally,  the  mean  scores  for  each  profile  across  respondents  were  subjected  to  a 
paired  sample  T-test  to  provide  a  second  look  at  possible  ratee  gender  effects  that  did  not 
show  up  during  the  regression.  The  paired  sample  test  was  possible  because  there  were 
24  profiles  on  both  surveys  that  had  the  same  behavior  order  and  behaviors,  but  1 2  had 
female  names  and  1 2  had  male  names. 
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IV.  Results 


Demographic  Information 

Tables  4  through  8  provide  the  sample  demographics.  Note  that  four  Colonels 
were  included  in  the  sample.  Either  they  filled  out  the  survey  for  a  missing  respondent, 
or  were  promoted  to  Colonel  and  the  computer  database  that  generated  the  sample  had 
not  yet  been  updated.  From  Table  6  you  can  see  that  female  respondents  were 
underrepresented  in  the  results.  This  resulted  from  the  random  nature  of  the  sample 
selection.  Females  were  underrepresented  in  the  sample  selected,  but  had  a  response  rate 
comparable  to  the  overall  response  rate. 


Table  4.  Survey  Type 


Normal 

Version 

Reversed 

Version 

Number  Returned 

94 

115 

Percentage 

45% 

55% 

Table  5.  Rank  of  Respondents 


2nd  Lt 

1st  Lt 

Captain 

Major 

Lt  Colonel 

Colonel 

Number  Returned 

6 

13 

73 

67 

46 

4 

Percentage 

3% 

6% 

35% 

32% 

22% 

2% 

Table  6.  Gender  of  Respondents 


Male 

Female 

Number  Returned 

186 

23 

Percentage 

89% 

11% 

Air  Force  Percentage 

83% 

17% 
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Table  7.  Job  Category  of  Respondents 


Analyst/Engineer 

Medical 

Nonrated  Ops 

Rated 

Support 

Other 

Number  Returned 

25 

7 

16 

31 

128 

2 

Percentage 

12% 

3% 

8% 

15% 

61% 

1% 

Table  8.  Commissioning  Source  of  Respondents 


Academy 

OTS 

ROTC 

Other 

Number  Returned 

36 

59 

100 

14 

Percentage 

17% 

28% 

48% 

7% 

Air  Force  Percentage 

21% 

21% 

44% 

14% 

Personality  Items  and  PANAS — Correlation,  Reliability,  and  Factor  Analyses 

The  correlation  of  the  personality  and  PANAS  items  indicated  that  a  reliability 
analysis  could  be  done  and  would  probably  generate  adequate  results.  Most  of  the 
correlations  between  items  designed  to  measure  the  same  facets  or  factors  were  greater 
than  .4  and  all  but  one  were  significant  at  the  p<  .01  level.  That  one  was  significant  at  the 
p  <.05  level. 

Table  9  lists  the  personality  facets  with  their  numerical  reference  (C1-C6).  The 
reliability  analysis  turned  out  very  favorable  results.  The  personality  facets  were  very 
similar  to  published  IPIP  results  with  the  exception  of  C6.  These  reliabilities  also 
compared  favorably  to  results  achieved  with  the  NEO  FFI.  Table  9  reports  the 
coefficient  alpha  of  each  personality  facet:  Cl,  C2,  C3,  C4,  C5,  and  C6.  It  also  reports 
the  published  reliability  of  the  IPIP  and  Costa  and  McRae’s  NEO  PI  (1992)  for 
comparison.  The  results  indicate  that  the  items  can  be  combined  into  summated  scales 
that  are  reasonably  reliable.  The  reliability  of  the  facets  with  respect  to  the  factor 
Conscientiousness  was  also  very  favorable. 
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Table  9.  Reliability  Analysis  Results  for  Conscientiousness  Factor 


Personality  Facet 

Coefficient 

Alpha 

Published 
IPIP  Alpha 

Published 
NEO  PI  Alpha 

Cl -Self-Efficacy 

.8456 

.78 

.70 

C2-Orderliness 

.8338 

.82 

.74 

C  3 -Dutifulness 

.7201 

.71 

.67 

C4-Achievement  Striving 

.8335 

.78 

.67 

C5-Self-Discipline 

.8654 

.85 

.80 

C6-Cautiousness 

.5393 

.76 

.70 

Summated  Scale 

.7158 

.80 

.75 

n=209 

The  first  six  facets  in  Table  9,  representing  one  personality  factor,  were  submitted 
to  a  factor  analysis.  The  analysis  used  Principle  Component  Analysis  with  a  Varimax 
rotation.  The  Varimax  Principal  Component  Analysis  rotation  was  chosen  because  it  is 
the  method  used  by  the  research  that  developed  this  type  of  five-factor  personality  model 
(Costa  and  McRae,  1992  and  Goldberg,  1999).  Only  two  factors  were  extracted  with 
eigenvalues  of  3.097  and  1.021.  As  you  can  see  in  Table  10,  C2  and  C6  loaded 
significantly  against  the  second  component  and  C3  exhibited  significant  cross  loadings. 
For  this  reason,  a  manual  extraction  of  factors  was  also  performed.  In  Table  1 1 ,  you  can 
see  that  the  facets  did  load  at  a  high  level  against  the  one  component.  Because  of  this 
complication,  however,  all  further  analysis  examined  these  personality  constructs  at  both 
the  individual  facet  level  and  at  the  factor  level. 
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Table  10.  Factor  Analysis  Varimax  Rotation 


Component 

1 

2 

Cl -Self-Efficacy 

.879 

1.928E-02 

C2-Orderliness 

8.612E-02 

.797 

C3-Dutifulness 

.529 

.515 

C4-Achievement  Striving 

.899 

^  .208 

C5-Self-Discipline 

.731 

.377 

C6-Cautiousness 

.189 

.771 

_ _ _ _  _ n=209 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Varimax  with  Kaiser  Normalization, 
a.  Rotation  converged  in  3  iterations. 


Table  11.  Factor  Analysis  Manual  Extraction 


Component 

1 

Cl -Self-Efficacy 

.737 

C2-Orderliness 

.520 

C3 -Dutifulness 

.727 

C4-Achievement  Striving 

.860 

C5-Self-Discipline 

.816 

|C6-Cautiousness 

.591 

n=209 

Extraction  Method:  Principal  Component  Analysis, 
a.  One  component  extracted. 


The  PANAS  measures  were  also  highly  reliable  with  coefficient  alphas  of  .9030 
for  PA  and  .8726  for  NA.  Table  12  shows  the  factor  analysis  for  the  measures.  This  was 
a  manual  extraction  of  two  factors  using  Principle  Component  Analysis  and  a  Varimax 
rotation.  The  factors  had  eigenvalues  of  5.430  and  4.797.  The  free  extraction  resulted  in 
two  further  factors  with  eigenvalues  of  1.336  and  1.041,  but  they  were  not  as  clean  as  the 
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first  two  components.  It  appears  that  these  measures  can  be  reliably  combined  into  their 
respective  scales,  PA  and  NA,  for  use  in  further  analysis. 


Table  12.  Factor  Analysis  PANAS  Scales,  Manual  Extraction 


Component 

1 

2 

PA1 

.751 

-8.158E-02 

PA2 

.713 

.176 

PA3 

.604 

1.666E-02 

PA4 

.812 

-1.701E-02 

PA5 

.748 

-4.638E-02 

PA6 

.689 

-.138 

PA7 

.111 

9.020E-03 

PA8 

.770 

1.744E-02 

PA9 

.767 

3.832E-02 

PA  10 

.679 

-4.620E-02 

NA1 

-3.490E-02 

.686 

NA2 

-8.551E-02 

.709 

NA3 

6.350E-02 

.627 

NA4 

-2.207E-02 

.744 

NA5 

-8.732E-02 

.614 

NA6 

-3.336E-02 

.690 

i  xr&m 

2.880E-02 

.576 

-4.067E-02 

NA9 

8.991E-02 

.708 

NA10 

3.797E-02 

.781 

n=209 

Rotated  Component  Matrix. 

Extraction  Method:  Principal  Component  Analysis. 
Rotation  Method:  Varimax  with  Kaiser  Normalization, 
a.  Rotation  converged  in  3  iterations. 


The  First  Level  Regression  of  Each  Respondent’s  Profiles 


The  analysis  of  the  data  resulted  in  229  first  level  regressions  with  the  resulting 
R2,  F  scores,  and  beta  weights.  There  were  18  cases  excluded  from  analysis  because  the 
R2  value  was  below  .5,  the  result  of  2  or  more  insignificant  beta  weights  for  the  behavior 
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dimensions.  Two  other  cases  were  excluded  because  the  respondents  had  one  or  more 
negative  beta  weights,  which  has  no  logical  basis.  The  order  terms,  gender  dummy 
variable,  and  interaction  terms  were  insignificant  for  most  of  the  respondents.  The 
minimum  and  maximum  terms,  considered  together,  were  insignificant  for  most  of  the 
respondents.  Therefore,  the  final  regression  reported  here  is  the  result  of  the  following 
equation:  op  =  J30  +  (3ld  (x, )  +  /3lp  (x2 )  +  j8:/  (x3 )  +  J3jd  (x4 ) .  The  R  values  are  reported  in 

Table  34  in  Appendix  E. 

Results  for  the  Individual  Behavior  Ratings 

First,  descriptive  statistics  of  each  of  the  four  dimensions’  individual  behavior 
ratings  was  performed  and  is  reported  in  Table  13.  For  simplicity,  leadership,  task 
performance,  interpersonal  facilitation,  and  job  dedication  will  be  referred  to  from  this 
point  as  LD,  TP,  IF,  and  JD  respectively.  All  behavior  ratings  in  each  dimension  have 
relatively  low  kurtosis  and  skewness.  The  means  are  also  very  close  to  each  other,  which 
was  one  of  the  objectives  of  the  behavior  combinations.  Note  that  LD  had  less  variance 
than  IF  and  JD,  but  more  than  TP.  Tables  14  through  17  display  the  mean  of  each 
behavior  used  in  this  research.  The  behaviors  are  ranked  by  means  in  descending  order 
for  each  behavior  dimension.  The  sample  size  for  the  computations  was  n=229. 

Although  20  respondents  were  removed  from  the  regression  analysis  for  the  stated 
reasons,  there  was  no  logical  basis  for  any  removals  from  the  means  analysis.  The 
regression  required  statistically  meaningful,  logical  beta  weights,  but  there  was  no  reason 
to  invalidate  the  individual  behavior  ratings. 
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Table  13.  Descriptive  Statistics  for  the  Individual  Behavior  Dimension  Ratings 


LD 

TP 

IF 

JD 

Mean 

5.15 

Mean 

5.19 

Mean 

4.934 

Mean 

5.22 

Standard  Error 

0.02 

Standard  Error 

0.02 

Standard  Error 

0.017 

Standard  Error 

0.02 

Median 

5 

Median 

5 

Median 

5 

Median 

5 

Mode 

5 

Mode 

5 

Mode 

5 

Mode 

6 

SD 

1.18 

SD 

1.15 

SD 

1.19 

SD 

1.24 

Variance 

1.40 

Variance 

1.32 

Variance 

1.42 

Variance 

1.53 

Kurtosis 

0.55 

Kurtosis 

-0.18 

Kurtosis 

0.23 

Kurtosis 

0.57 

Skewness 

-0.59 

Skewness 

-0.29 

Skewness 

-0.45 

Skewness 

-0.67 

Range 

6 

Range 

6 

Range 

6 

Range 

6 

Minimum 

1 

Minimum 

1 

Minimum 

1 

Minimum 

1 

Maximum 

7 

Maximum 

7 

Maximum 

7 

Maximum 

7 

Sum 

25708 

Sum 

25913 

Sum 

24631 

Sum 

26049 

n 

4992 

n 

4992 

n 

4992 

n 

4992 
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Table  14.  Leadership  Behavior  Means 


Behavior 

Means 

Uses  good  judgment  in  decision  making 

6.193 

Reacts  confidently  when  the  unexpected  occurs 

5.789 

Recognizes  and  encourages  effective  performance 

5.721 

Resolves  conflicting  organizational  demands 

5.546 

Guides  and  directs  subordinates  effectively 

5.541 

Keeps  subordinates  focused  on  mission  requirements 

5.496 

Works  to  create  an  effective  unit  atmosphere 

5.426 

Assigns  subordinates  duties  and  responsibilities  appropriate  for  their  abilities 

5.345 

Resolves  conflicts  between  members  of  the  unit 

5.328 

Provides  appropriate  feedback  to  subordinates 

5.277 

Supports  subordinates 

5.242 

Makes  tough  decisions  quickly 

5.242 

Ensures  deadlines  and  performance  standards  are  met  1 

5.227 

Persuades  others  both  inside  and  outside  the  organization 

5.192 

Coordinates  subordinates'  efforts  to  minimize  conflicts  1 

5.146 

Represents  the  group  effectively 

5.055 

Takes  a  position  on  controversial  issues 

5.013 

Speaks  effectively 

4.993 

Behaves  consistently  with  subordinates 

4.873 

Monitors  the  status  of  work  in  progress 

4.603 

Maintains  high  visibility  both  on  and  off  the  iob 

4.539 

Avoids  trespassing  on  others'  responsibility  areas 

4.320 

n=229 
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Table  15.  Task  Performance  Behavior  Means 


Behavior 

Means 

Solves  urgent,  unexpected  problems  expertly 

6.175 

Solves  technical  problems  expertly 

5.766 

Anticipates  potential  problems 

5.739 

Accomplishes  job  tasks  expertly 

5.729 

Troubleshoots  expertly 

5.643 

Provides  expert  technical  advice  to  others 

5.483 

Uses  technical  expertise  to  meet  real  world  needs 

5.432 

Communicates  task  information  effectively 

5.408 

Writes  clearly  and  concisely 

5.335 

Performs  specialized  tasks  skillfully 

5.333 

Prioritizes  work  tasks  efficiently 

5.330 

Collects  and  accurately  interprets  information 

5.159 

Keeps  up  with  the  newest  technology 

5.039 

Plans  and  organizes  work 

4.893 

Provides  others  with  current  technical  information 

4.865 

Operates  equipment  skillfully 

4.764 

Uses  equipment,  tools,  and  computers  proficiently 

4.690 

Performs  routine  tasks  efficiently 

4.644 

Uses  technical  material  effectively 

4.548 

Performs  safely 

4.223 

n-229 
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Table  16.  Interpersonal  Facilitation  Behavior  Means 


Behavior 

Means 

Praises  coworkers  when  they  are  successful 

5.517 

Helps  someone  without  being  asked 

5.480 

Talks  to  others  before  taking  actions  that  affect  them 

5.474 

Develops  and  maintains  good  working  relationships 

5.421 

Listens  to  others'  ideas  about  getting  work  done 

5.421 

Encourages  others  to  overcome  differences  and  get  along 

5.384 

Voluntarily  pitches  in  to  help  the  group 

5.328 

Encourages  coworkers  to  stick  together  in  hard  times 

5.317 

Cooperates  with  others  in  the  team  effectively 

5.308 

Lends  a  hand  when  a  coworker  needs  it 

5.306 

Supports  or  encourages  coworkers 

5.247 

Displays  concern  for  others 

5.153 

Seeks  others  opinions 

5.127 

Offers  to  help  others  do  their  work 

5.035 

Displays  a  cheerful,  confident  outlook 

5.035 

Coordinates  actions  with  others 

5.022 

Shows  respect  for  others 

5.000 

Treats  others  fairly 

4.924 

Says  things  to  reduce  conflicts 

4.766 

Says  things  to  make  people  feel  good  about  themselves 

4.731 

Offers  friendly  advice 

4.493 

Gives  coworkers  advice  about  how  to  do  their  jobs 

4.487 

Acts  courteously 

4.417 

Gets  along  with  others 

4.404 

Acts  warm  and  sociable 

4.253 

Avoids  arguments 

3.299 

n=229 
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Table  17.  Job  Dedication  Behavior  Means 


Behavior 

Means 

Asks  for  challenging  work  assignments 

5.784 

Strives  to  excel 

5.742 

Takes  the  initiative  to  solve  a  work  problem 

5.738 

Volunteers  for  difficult  assignments 

5.712 

Tackles  a  difficult  work  assignment  enthusiastically 

5.642 

Adapts  to  difficult  conditions 

5.603 

Overcomes  obstacles  to  complete  a  task 

5.568 

Pays  close  attention  to  important  details 

5.535 

Takes  responsibility  for  his/her  actions 

5.526 

Ensures  work  is  done  right 

5.520 

Puts  extra  effort  into  a  task 

5.415 

Puts  in  extra  hours  to  get  work  done  on  time 

5.402 

Overcomes  hardships 

5.271 

Gives  up  personal  time  for  the  mission 

5.236 

Volunteers  for  additional  duties 

5.218 

Defends  the  supervisor's  decisions 

5.114 

Avoids  shortcuts  when  work  is  overdue 

4.987 

Performs  consistently  and  reliably 

4.980 

Works  hard 

4.856 

Complies  with  instructions  even  when  supervision  is  not  present 

4.819 

Shows  respect  for  authority 

4.801 

Displays  proper  military  appearance  and  bearing 

4.491 

Follows  the  supervisor's  instructions 

4.410 

Renders  proper  military  courtesy 

4.297 

n=229 

The  individual  behavior  means  from  the  final  survey  respondents  were  tested  to 
see  if  they  were  statistically  the  same  as  the  means  found  in  the  pilot  study  and  the  means 
found  by  Hurry  (1995).  A  two-tailed,  paired  sample  analysis  indicated  that  the  means 
were  statistically  different  from  Hurry’s  means  (p=. 002)  and  different  from  the  pilot 
study’s  means  (p=. 001).  The  final  survey  individual  behavior  means  were  not 
significantly  correlated  with  Hurry’s  (1995)  means,  or  with  the  pilot  survey  means  (two 


50 


tailed  test,  alpha=05).  The  means  comparison  results  are  summarized  in  Table  18,  while 
the  correlation  table  is  shown  in  Table  19. 


Table  18.  Means  Comparison  Summary  Table 


Means  Comparison 

p-value 

Survey 

Means 

Final  Means  to  Pilot  Survey  Means 

.001* 

Hurry’s 

5.40 

Final  Means  to  Hurry’s  Means 

.002* 

Pilot 

4.92 

““Indicates  statistically  significant  difference 

Final 

5.16 

n=92 

Table  19.  Correlation  Table  of  Means 


Final  Means 

Hurry  Means 

Pilot  Means 

Final  Means 

1 

Hurry  Means 

-0.113287882 

1 

Pilot  Means 

0.004491415 

0.031072635 

1 

Alpha  =.05 

All  correlations  are  statistically  insignificant. 
n=92 

In  an  effort  to  confirm  the  reliability  of  the  behavior  measures,  two  random  sub 
samples  (n=50)  were  pulled  from  the  final  survey  data  for  comparison.  Table  20  shows 
the  results  of  a  comparison  of  the  sub  samples’  behavior  means.  As  you  can  see,  the 
paired  sample  comparison  shows  the  means  are  statistically  different  from  each  other,  but 
the  samples  are  highly  correlated.  The  number  of  observations  indicates  the  number  of 
behaviors,  while  the  number  of  original  respondents  represented  in  these  sub  samples  is 
50.  Table  21  shows  the  correlation  between  the  two  sub  samples  was  .963  and  this 
correlation  is  significant  at  p=.01 . 


51 


Table  20.  Final  Survey  Sub  Sample  Comparison  of  Means 


Sample  1  Means 

Sample  2  Means 

Mean 

5.135122306 

5.236624179 

Variance 

0.228140284 

0.220689517 

Observations 

92 

92 

Pearson  Correlation 

0.962789072 

Hypothesized  Mean 

Difference 

0 

df 

91 

tStat 

-7.520020973 

P(T<=t)  one-tail 

1.86544E-1 1 

t  Critical  one-tail 

1.661771876 

P(T<=t)  two-tail 

3.73087E-1 1 

t  Critical  two-tail 

1.986377356 

Table  21.  Sub  Sample  Comparison  Correlation  Table 


Sample  1  Means 

Sample  2  Means 

Sample  1  Means 

1 

Sample  2  Means 

0.963* 

1 

*  Significant  at  p=.01 . 

Results  for  Differences  in  the  Relative  Importance  of  Performance  Dimensions 

First,  descriptive  statistics  of  each  of  the  four  dimension’s  beta  weights  were 
performed.  The  descriptive  statistics  are  reported  in  Table  22.  Note  that  although  they 
all  have  very  low  variance,  IF  has  the  most  variance,  .009.  All  of  the  dimensions  display 
positive  kurtotic  tendencies.  These  results  explain  why  IF  was  the  only  dimension  to 
pass  the  test  for  normality,  discussed  later.  It  has  the  least  kurtosis  and  skewness  and  also 
has  the  greatest  variance.  This  will  necessitate  a  nonparametric  examination  of  the  data. 
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Table  22.  Descriptive  Statistics  of  the  Four  Dimensions’  Beta  Weights  from 
Regression  on  Overall  Performance 


LD 

TP 

IF 

JD 

Mean 

0.282 

Mean 

0.257 

Mean 

0.254 

Mean 

0.2679 

Standard  Error 

0.006 

Standard  Error 

0.006 

Standard  Error 

0.006 

Standard  Error 

0.006 

Median 

0.282 

Median 

0.255 

Median 

0.254 

Median 

0.266 

Mode 

0.313 

Mode 

llaa 

Mode 

EBB 

Mode 

0.288 

SD 

0.088 

SD 

SD 

ebb 

SD 

0.084 

Variance 

0.007 

Variance 

0.007 

Variance 

0.009 

Variance 

0.007 

Kurtosis 

6.229 

Kurtosis 

1.687 

Kurtosis 

0.894 

Kurtosis 

2.179 

Skewness 

1.058 

Skewness 

0.436 

Skewness 

0.058 

Skewness 

0.384 

Range 

0.81 

Range 

0.562 

Range 

0.599 

Range 

0.638 

Minimum 

0.003 

Minimum 

0 

Minimum 

0 

Minimum 

0 

Maximum 

0.813 

Maximum 

0.562 

Maximum 

0.599 

Maximum 

■ilacra 

Sum 

59.059 

Sum 

53.836 

Sum 

53.091 

Sum 

n 

209 

n 

209 

n 

209 

n 

WE3L 

Table  34  in  Appendix  E  also  reports  the  beta  weights  for  each  respondent.  Only 
52  out  of  916  individual  beta  weights  were  not  significant  at  the  p<.05  level.  These 
insignificant  beta  weights  will  be  included  in  this  analysis  because  the  insignificance  of 
these  beta  weights  means  that  they  cannot  be  determined  significantly  different  from 
zero.  These  beta  weights  were  nearly  zero,  so  their  inclusion  in  the  analysis  makes  sense 
The  exception  for  the  inclusion  of  the  beta  weights  will  be  the  exclusion  of  regressions 
with  R2  values  below  .5.  These  regressions  had  2  or  more  insignificant  beta  weights  and 
represented  poor  fitting  relative  to  the  rest  of  the  sample. 

These  beta  weights  will  be  used  in  the  second  set  of  regression  analyses.  But 
first,  a  comparison  of  means  is  necessary  to  test  one  of  the  hypotheses  for  this  research. 
Table  23  includes  the  mean  beta  weights  for  the  entire  sample  and  various  sub  samples. 
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Using  Excel’s  two-sample  F  test  for  equal  variances  and  Levene’s  test  for  equal  variance, 
the  beta  weights  all  have  statistically  significant  equal  variances  (alpha=. 05).  The 
analysis  found  that  there  was  no  difference  in  the  rating  policies  of  officers  when 
considering  gender,  job  category,  or  commissioning  source.  There  were  some  subtle 
differences  in  rating  policies  by  rank,  however. 


Table  23.  Mean  Beta  Weights  for  Behavior  Dimensions  of  Sample  and  Various  Sub 
Samples  Derived  from  Regression  on  Overall  Performance 


n 

LD 

TP 

IF 

JD 

Entire  Sample 

209 

0.283* 

0.258 

0.254 

0.268 

Majors  and  Above 

117 

0.293* 

0.257 

0.252 

0.271 

Captains  and  Below 

92 

0.269 

0.259 

0.257 

0.264 

AFIT  Students 

52 

0.277 

0.255 

0.278 

0.265 

DAU  Students 

21 

0.275 

0.247 

0.243 

0.275 

Mail  Respondents 

136 

0.286* 

0.260 

0.247 

0.268 

Notes:  All  tests  were  two-tailed  tests 
The  significance  level  for  all  te 
*  Statistically  different  from  TP  and  IF 
All  other  comparisons  were  statistical! 

for  equal  means. 

:sts  was  alpha=.05. 
and  statistically  same  as  JD. 
y  the  same. 

For  the  entire  sample,  LD  was  statistically  the  same  as  JD  and  more  important 
than  TP  and  IF  at  the  p<.05  level,  although  the  p  value  for  the  LD,  JD  comparison  was 
.08.  JD,  TP,  and  IF  were  statistically  the  same.  For  Majors  and  above,  the  same 
relationship  as  before  held,  although  the  p  value  for  the  LD,  JD  comparison  was  .07.  The 
other  three  dimensions  were  the  same.  For  Captains  and  below,  all  four  dimensions  were 
statistically  the  same  at  the  alpha=.05  level.  For  AFIT  students,  all  four  dimensions  were 
statistically  the  same  at  the  alpha=.05  level. 
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The  DAU  students  also  felt  that  all  four  dimensions  were  statistically  the  same  at  the 
alpha=. 05  level.  Finally,  mailed  survey  respondents  felt  that  LD  was  the  same  as  JD,  but 
different  from  the  other  two  dimensions.  JD,  IF,  and  TP  were  all  statistically  the  same. 

The  data  is  also  approximately  normally  distributed  in  the  normal  probability  plot 
generated  in  SPSS,  but  only  interpersonal  facilitation  passed  the  Kolmogrorov-Smirnov 
test  of  normality.  SPSS  considers  .2  to  be  the  lower  bound  of  significance  in  this  test, 
and  interpersonal  facilitation’s  significance  was  .2.  Table  24  shows  the  test  of  normality 
results. 


Table  24.  Test  of  Normality 


Kolmogorov-Smirnov 

Statistic 

df 

Sig. 

LD 

.060 

209 

.061 

TP 

.081 

209 

.002 

IF 

.054 

209 

.200 

JD 

.073 

209 

.009 

*  This  is  a  lower  bound  of  the  true  significance. 
Lilliefors  Significance  Correction 


Because  the  four  performance  dimensions  tested  so  poorly  for  normality,  it  was 
necessary  to  perform  nonparametric  statistics  to  verify  the  results  found  on  the  two-tailed 
test  between  dimensions.  The  Tukey-Kramer  analysis  was  chosen  and  performed  in 
JMPIN  with  alpha=.05.  The  Compare  All  Pairs  method  was  chosen  to  achieve  an  exact 
alpha-level  test  since  this  research  has  equal  sample  sizes.  The  results  mirrored  the  two- 
tailed  results  for  the  entire  sample.  LD  was  significantly  greater  than  TP  and  IF,  but  the 
same  as  JD.  JD,  IF,  and  TP  were  not  statistically  different  from  each  other.  This  is 
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denoted  in  the  bottom  half  of  Table  25  by  the  positive  values  at  the  intersections  LD/TP 
and  LD/IF. 


Table  25.  Tukey-Kramer  HSD  Means  Comparisons 


Dif=Mean  [i]-Mean  [j] 

LD 

JD 

TP 

IF 

LD 

0.000000 

JD 

-0.01468 

0.000000 

TP 

-0.02499 

-0.01031 

0.000000 

IF 

-0.02856 

-0.01388 

-0.00356 

0.000000 

Abs  (Dif)-LSD** 

LD 

JD 

TP 

IF 

LD 

-0.02183 

JD 

-0.00715 

-0.02183 

TP 

0.003160** 

-0.01152 

-0.02183 

IF 

0.006724** 

-0.00796 

-0.01827 

-0.02183 

Alpha=0.05  and  q*=2. 57429 
Comparisons  for  all  pairs  using  Tukey-Kramer  HSD 
**Positive  values  show  pairs  of  means  that  are  significantly  different. 


Another  necessary  comparison  is  how  different  groups  felt  about  a  certain 
dimension.  An  examination  of  the  data  shows  that  higher-ranking  officers’  mean 
leadership  beta  weight  was  significantly  higher  than  the  lower-ranking  officers’  mean 
leadership  beta  weight  at  the  p<.05  level.  There  are  other  differences  in  the  different  sub 
samples  between  AFIT,  DAU,  and  mail  respondents,  but  they  can  also  be  explained  by 
the  difference  between  the  ranks  because  the  AFIT  and  DAU  students  were,  with  a  few 
exceptions,  Captains  and  below,  while  the  mail  respondents  were  comprised  of  mostly 
Majors  and  above.  This  will  be  discussed  further  in  Chapter  V. 


Second  Level  Regression 

The  second  level  regression  uses  each  of  the  unstandardized  beta  weights  as 
dependent  variables,  in  turn.  All  of  the  demographic  items,  personality  facets,  and 
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PANAS  were  independent  variables.  None  of  these  regressions  produced  significant 
results.  This  analysis  was  done  using  a  stepwise  regression  and  no  variables  were  entered 
with  a  probability  of  F  to  enter  of  .05  and  remove  of .  1 0.  Correlation  and  covariance 
matrices  were  produced  to  examine  the  feasibility  of  analysis  using  SEM  methodology, 
but  there  were  only  two  significant  correlations  between  the  beta  weights  and  any  of  the 
other  variables.  (3id  was  correlated  with  C2,  Orderliness,  at  the  .05  level.  This  correlation 
was  .136.  Also,  Positive  Affectivity  was  negatively  correlated  with  ptp  at  the  .05  level. 
This  correlation  was  -.134.  The  covariance  matrix  showed  covariance  between  beta 
weights  and  all  other  variables  was  nearly  0.  Therefore,  a  SEM  analysis  was  not 
performed. 

Consistency  Results 

A  stepwise  regression  was  used  for  this  analysis  with  the  same  probability  of  F  to 
enter  and  remove  as  previously  mentioned.  A  significant  model  was  not  found.  A 
further  analysis  was  performed,  guided  by  the  results  achieved  in  the  pilot  survey.  The 
pilot  survey  found  a  significant  regression  model  with  the  variables  PA,  Total  Active 
Federal  Military  Service  years,  and  C6  (Cautiousness)  predicting  rater  consistency. 

The  pilot  survey  respondents  were  almost  exclusively  Majors  and  above,  so  the 
final  survey  data  was  reexamined  using  only  Majors  and  above  from  the  full  survey 
respondents.  The  SPSS  output  tables  are  shown  in  Tables  26  through  28.  This  is  the 
final  model  after  all  other  insignificant  variables  have  been  removed.  The  results  were 
weak  but  significant,  with  an  R2  value  of  .041  and  an  F  score  of  4.926.  This  result  is 
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similar  to  that  found  in  the  pilot  study.  TAFMS  represents  the  total  time  a  respondent  has 
been  in  the  Air  Force. 

Table  26.  Model  Summary  for  Consistency  Regression 


Model 

R 

R  Square 

Adjusted 
R  Square 

Std.  Error  of 
the  Estimate 

1| 

.203 

.041 

.033 

9.2923E-02 

a.  Predictors:  (Constant),  TAFMS 


Table  27.  ANOVA  Table  for  Consistency  Regression 


Model 

Sum  of  Squares 

df 

Mean 

Square 

F 

Sig. 

1 

Regression 

4.254E-02 

1 

4.254E-02 

4.926 

.028 

Residual 

.993 

115 

8.635E-03 

Total 

1.036 

116 

a.  Predictors:  (Constant),  TAFMS 

b.  Dependent  Variable:  r_squared 


Table  28.  Coefficients  Table  for  Consistency  Regression 


Unstandardized 

Coefficients 

Standardized 

Coefficients 

t 

Sig. 

Model 

B 

Std.  Error 

Beta 

1 

(Constant) 

.676 

.034 

19.682 

.000 

TAFMS 

4.379E-03 

.002 

.203 

2.220 

.028 

a.  Dependent  Variable:  r  squared 


Leniency  Results 

The  mean  overall  importance  for  each  respondent  was  also  examined  to  look  for 
relationships  between  rater  leniencies  and  the  personality  facets,  PA  or  NA.  There  were 
five  significant  relationships  between  average  overall  performance  and  C3,  C4,  C5, 
Conscientiousness,  and  PA.  These  are  shown  in  Table  29. 
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Table  29.  Significant  Correlations  Between  Average  Overall  Performance  and 

Personality  Measures 


C3 

C4 

C5 

CON 

PA 

AVG  OP 

Pearson  Correlation 

.164* 

.249** 

.252** 

.245** 

.228** 

Sig.  (2-tailed) 

.017 

.000 

.000 

.000 

.001 

n 

209 

209 

209 

209 

209 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed). 
*  Correlation  is  significant  at  the  0.05  level  (2 -tailed). 


Ratee  Gender  Effects  Results 


The  ratee  gender  was  entered  into  the  first  level  regression  as  a  dummy  variable 
and  no  significant  effect  was  found.  However,  the  survey  was  designed  to  allow  a  paired 
sample  comparison  of  means  of  the  ratings  received  across  respondents  for  overall 
performance  and  the  four  performance  dimensions.  There  were  no  significant  mean 
differences  at  the  p=  .05  level  when  these  means  were  compared  using  a  two-tailed  paired 
mean  t-test.  The  reported  p  value  was  .81 .  The  reported  mean  for  female  profiles  was 
5.142,  while  the  reported  mean  for  male  profiles  was  5.128. 
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V.  Conclusions  and  Discussions 


Relative  Importance  of  Behavior  Dimensions  to  Overall  Performance 

The  results  suggest  that  the  order  effect  found  in  Hurry’s  1995  research  resulted 
in  spurious  results.  However,  leadership  is  still  the  most  important  performance 
dimension  by  a  narrow  margin.  This  does  not  mean  that  junior  officers  should  focus  on 
leadership  behaviors,  which  is  discussed  in  greater  detail  in  “Weaknesses.”  It  does  mean 
that  the  Air  Force  organizational  norm  is  that  leadership  behavior  is  slightly  more 
important  than  other  behavior  dimensions  in  determining  overall  performance.  Training 
programs,  therefore,  should  perhaps  examine  their  curriculum  to  see  if  this  importance  is 
recognized.  If  it  is  not,  then  a  policy  decision  should  be  made  to  decide  if  leadership 
should  be  emphasized  more  than  the  other  behavior  dimensions.  If  the  slightly  greater 
importance  of  leadership  is  recognized  in  the  training  curriculum,  perhaps  this  survey 
merely  captured  the  results  of  an  effective  training  program. 

The  first  level  regression  resulted  in  very  clean  results  with  nearly  all  behavior 
dimensions  having  individual  significance  in  the  model.  The  interpretation  of  this  result 
is  that  each  respondent  had  his  or  her  own  priority  about  which  behavior  dimensions  were 
most  important  and  this  priority  showed  up  in  the  regression.  When  the  entire  sample 
was  considered,  however,  the  individual  priorities  canceled  out  for  task  performance, 
interpersonal  facilitation,  and  job  dedication,  leaving  leadership  as  the  one  different 
dimension.  The  final  equation  used  to  predict  overall  performance  with  the  four 
performance  dimensions  is  OP  =  /?,  +  .283 (Id)  +  .258 (tp)  +  ,254(//)  +  .268 (jd) .  An 
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example  of  this  equation  in  action  would  be  to  consider  a  junior  officer  who  scored  a  4,  6, 
5,  and  7  in  leadership,  task  performance,  interpersonal  facilitation,  and  job  dedication 
respectively.  If  that  junior  officer  were  evaluated  with  this  equation,  his/her  overall 
performance  would  be  rated  a  5.8. 

Interestingly,  officers  in  the  grade  of  Majors  and  above  had  a  different  policy  than 
company  grade  officers  (CGOs).  The  officers  who  actually  rate  junior  officers  felt  that 
leadership  was  the  most  important  dimension,  an  important  discovery.  CGOs,  however, 
felt  that  all  of  the  dimensions  were  equally  important.  This  disparity  identifies  a  curious 
situation.  The  CGOs  of  today  are  the  Majors  and  above  of  tomorrow.  This  gap  could  be 
the  product  of  a  change  in  the  Air  Force  culture  that  has  not  caught  up  to  the  senior  ranks 
yet,  or  it  could  be  that  the  survey  found  a  change  in  policy  that  takes  place  in  officers  as 
they  gain  in  rank  and  experience  and  receive  more  training. 

Another  important  finding  in  these  results  is  that  order  effects  create  spurious 
results  in  an  otherwise  good  policy  capturing  effort.  Policy  capturing  studies  should  be 
careful  to  randomize  within  profiles  and  report  the  randomization  to  prevent  order  effects 
and  evade  criticism.  This  study’s  results  differed  significantly  from  Hurry’s  results,  even 
though  both  found  that  leadership  was  the  most  important  dimension,  and  any  research 
that  does  not  randomize  may  be  reporting  spurious  results. 

There  could  be  other  forces  at  work  to  create  the  result  found  in  this  research.  For 
example,  leadership  scores  could  have  had  more  variance  than  the  other  dimensional 
scores.  This  would  have  created  the  possibility  for  more  covariance,  resulting  in  a  better 
regression  result  and  higher  beta  weight  for  the  leadership  dimension.  Table  13  shows 
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that  this  is  not  the  case.  Leadership  had  less  variance  than  two  other  dimensions, 
interpersonal  facilitation  and  job  dedication.  Only  task  performance  had  lower  variance 
than  leadership. 

Individual  Behavior  Means 

There  were  also  some  consistency  problems  with  the  means  of  the  individual 
behaviors.  The  means  that  were  captured  with  the  final  survey  were  not  statistically  the 
same  as  those  captured  by  Hurry  (1995)  and  those  captured  in  the  pilot  survey.  The 
behavior  means  were  also  not  correlated  between  the  three  different  data  sets.  This  raises 
the  possibility  that  the  reliability  of  the  measures  is  not  adequate.  One  possible 
interpretation  of  this  is  that  the  behaviors  chosen  were  too  ambiguous  to  adequately 
capture  the  construct  of  interest.  Or,  the  behavior  may  have  been  mistakenly  thought  to 
represent  another  dimensional  construct  by  the  respondents. 

However,  two  sub  samples  were  pulled  and  compared  from  the  final  sample.  This 
comparison  yielded  favorable  results.  One  would  expect  that  the  means  might  be 
different,  but  if  the  items  are  measuring  the  same  things  reliably,  different  samples  should 
still  be  correlated.  The  sub  sample  comparison  indicated  that  these  measures  were 
reliable  across  this  particular  sample.  This  indicates  that  the  difference  between  the  final 
survey  means,  the  pilot  survey  means,  and  Hurry’s  means  may  be  the  result  of  something 
other  than  unreliable  measures.  For  example,  Hurry  measured  her  behaviors  in  a 
different  context  than  this  research  and  her  sample  was  less  representative  of  the  general 
Air  Force  because  it  was  limited  to  respondents  with  the  rank  of  Major  or  above  at 
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Wright-Patterson  Air  Force  Base.  The  pilot  survey  also  measured  Majors  and  above  at 
AFIT.  The  final  survey  was  a  much  closer  representation  of  the  Air  Force  and  utilized  a 
larger  sample.  Sub  samples  from  the  final  survey  indicated  that  the  measures  were 
reliable  within  the  sample.  Therefore,  I  conclude  that  the  final  survey  accurately 
measured  the  behaviors  in  the  context  provided  and  represents  a  better  measure  of  the 
total  Air  Force  officer  population’  views  on  the  value  of  these  behaviors  than  either  the 
pilot  survey  or  Hurry’s  survey. 

Second  Level  Regression  With  Demographic,  Personality,  and  PANAS  Variables 

The  second  level  regressions,  designed  to  find  whether  demographic,  personality, 
or  PANAS  variables  can  be  used  to  predict  the  unstandardized  beta  weights  from  the  first 
level  regression,  were  woefully  insignificant.  There  is  no  evidence  from  this  effort  to 
indicate  that  any  of  these  variables  can  be  used  to  predict  the  importance  of  a  behavior 
dimension.  The  correlation  results  showed  that  only  two  of  the  other  variables, 
Orderliness  and  PA,  were  correlated  with  the  beta  weights  and  the  covariance  matrix 
showed  that  there  was  little  or  no  covariance.  Without  correlation  and  some  level  of 
covariance,  a  SEM  analysis  of  the  data  would  be  useless. 

This  area  addresses  Hypothesis  3  through  6  and  Hypothesis  10  in  Chapter  II.  This 
study  found  that  the  mean  scores  for  the  four  dimensions  of  performance  do  not  vary  with 
rater  experience,  grade,  sex,  or  commissioning  source.  In  addition,  the  personality 
variables  do  not  explain  any  of  the  variance  in  the  dimensional  ratings.  This  does  not 
mean  that  personality  does  not  affect  the  rating  policy  of  Air  Force  raters,  only  that  this 
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study  was  not  able  to  find  any  relationship.  This  may  be  the  product  of  the  fictitious 
profiles.  It  would  seem  logical  that  personality  would  be  a  more  active  agent  in  policy  if 
there  were  an  interaction  between  the  rater  and  ratee.  This  method  precludes  this  because 
it  used  fictitious  profiles.  Junior  officers  should  not  think  that  personality  does  not 
matter,  however,  because  any  person  who  has  experienced  a  personality  clash  with  their 
supervisor  knows  that  it  can  affect  rating  decisions. 

Rater  Consistency 

As  expected,  rater  experience  was  a  statistically  significant  predictor  of  rating 
consistency  as  measured  by  adjusted  R  ,  but  only  for  officers  in  the  grades  Major  and 
above.  This  agrees  with  the  results  of  the  pilot  study  done  for  this  thesis,  but  the 
relationship  found  was  very  slight.  This  could  mean  that  raters  get  lackadaisical  as  they 
get  older  and  rate  everything  the  same,  or  it  could  mean  that  supervisors  are  more 
knowledgeable  about  their  policy  as  they  gain  more  experience.  The  relationship  was  so 
small,  though,  as  to  be  insignificant  in  a  practical  sense  because  experience  had  very  little 
influence  over  consistency.  The  rank  of  the  rater  and  the  number  of  officers  they 
currently  rate  were  insignificant  in  the  model,  but  both  of  these  variables  are  highly 
correlated  with  Total  Active  Federal  Military  Service  (TAFMS)  years.  They  are  highly 
correlated  because  the  longer  officers  are  in  the  Air  Force,  the  more  likely  they  are  to 
have  supervised  junior  officers.  This  is  evident  in  the  significant  correlation  between 
TAFMS,  the  number  of  officers  supervised,  and  years  of  supervisor  experience.  Because 
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of  this  correlation,  they  were  confounding  variables.  None  of  the  personality  variables 
had  explanatory  power  for  rater  consistency. 

Rating  Leniency 

Contrary  to  the  findings  of  Bernardin,  Cooke,  and  Villanova  (2000), 
Conscientiousness  was  positively  correlated  with  overall  performance,  indicating  that 
higher  levels  of  this  personality  factor  are  related  to  higher  overall  performance  scores. 
There  may  be  a  few  factors  at  work  here,  however.  First,  this  study’s  respondents  rated 
profiles  that  were  fictitious  while  Bernardin,  Cooke,  and  Villanova  were  measuring 
ratings  given  in  an  actual  setting  to  real  students.  Second,  this  research  measured  the  full 
range  of  the  personality  construct  offered,  using  60  item  measures.  Bernardin,  Cooke, 
and  Villanova  used  a  shortened  version  survey  instrument  with  10  items  for 
Conscientiousness. 

The  first  factor,  fictitious  versus  real,  is  a  limitation  of  this  study  and  may  explain 
part  of  the  reason  these  results  differed  from  Bernardin,  Cook,  and  Villanova  (2000). 
Since  these  profiles  were  fictitious,  the  respondents  may  not  have  felt  any  pressure  to 
control  the  level  of  the  ratings  because  there  were  no  rewards  for  accurate  ratings  or 
consequences  for  inaccurate  ratings  (Murphy  and  Cleveland,  1995,  242).  Therefore, 
respondents  may  let  their  natural  Positive  Affectivity  (the  positive  correlation  between 
PA  and  overall  performance)  influence  the  ratings  without  the  controls  of 
Conscientiousness  because  there  is  nothing  for  the  conscience  to  worry  about. 
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The  second  factor,  short  version  versus  full  version,  indicates  that  Bernardin, 
Cook,  and  Villanova’s  (2000)  findings  may  not  be  accurate  because  the  full  version  of 
the  personality  measures  is  a  better  measure  of  the  construct  of  interest,  in  this  case 
Conscientiousness.  A  fuller  measure  should  result  in  a  more  accurate  operationalization 
of  Conscientiousness  and  therefore  more  accurate  findings  based  on  the  measure. 
However,  this  researcher  would  have  found  a  differing  correlation  more  indicative  of  this 
difference  in  measures  rather  than  a  change  in  sign  indicating  a  completely  opposite 
relationship  than  that  found  by  Bernardin,  Cook,  and  Villanova  (2000).  This  should  be 
explored  further  outside  the  realm  of  policy  capturing  methodology  as  discussed  in 
“Future  Research.” 

Ratee  Gender  Effects 

The  gender  of  the  ratee  appears  to  have  no  effect  on  the  ratings  given  to  that 
particular  profile.  This  effect  was  examined  twice,  once  with  a  dummy  variable  with  full 
interaction  in  the  first  level  regression,  and  again  using  a  paired  sample  analysis 
comparing  the  means  of  the  female  profile  ratings  with  their  corresponding  identical  male 
counterparts.  Both  examinations  produced  negative  results.  There  is  no  gender  effect  in 
these  responses. 

Weaknesses 

This  study  has  several  weaknesses.  First  and  foremost,  although  part  of  the 
sample  selected  was  randomly  chosen  from  different  locations,  another  large  part  of  the 
sample  came  from  localized  students  attending  AFIT  and  DAU.  The  students  attending 
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AFIT  were  a  sample  of  convenience.  Some  demographics  were  controlled  to  provide  an 
even  spread  of  data  (i.e.  rank  and  AFSC),  but  the  other  variables  were  left  to  randomness 
and  were  close  to  the  Air  Force  percentages  reported  in  Chapter  IV.  So,  in  spite  of  this 
weakness,  the  sample  provides  a  fairly  representative  picture  of  the  Air  Force,  excluding 
consideration  of  the  variables  that  were  controlled.  An  effort  that  selected  respondents  in 
a  completely  random  nature  would  most  likely  be  more  generalizable  to  the  Air  Force 
than  this  effort. 

This  study  was  designed  to  identify  general  relative  importance  of  the  four 
dimensions  of  overall  performance.  It  does  not  capture  the  possibility  that  a  junior 
officer  can,  and  sometimes  does,  fill  a  position  in  the  Air  Force  that  requires  them  to 
display  a  higher  level  of  one  of  the  dimensions.  An  example  would  be  a  junior  captain 
who  has  been  assigned  as  a  branch  chief  and  supervises/leads  60  other  airmen.  The 
nature  of  his/her  position  requires  this  supervisor  to  place  more  importance  on  the 
leadership  behaviors  because  that  is  what  is  expected  of  that  job.  Therefore,  the  fictitious 
general  nature  of  the  profiles  means  that  this  research  could  not  be  used  to  predict  ratings 
of  a  real  junior  officer  with  a  particular  job. 

This  study  also  could  not  be  used  to  direct  junior  officers  in  their  performance 
under  a  particular  supervisor.  Each  respondent  felt  very  differently  about  the  four 
dimensions  of  overall  performance,  so  one  could  not  assume  that  a  supervisor  feels 
leadership  is  the  most  important  dimension.  This  study  can  inform  the  Air  Force, 
however,  that  leadership  is  considered  slightly  more  important  than  the  other  dimensions 
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in  determining  overall  performance,  and  this  can  be  used  to  guide  future  training  of 
junior  officers. 

The  study  was  also  limited  because  it  used  only  positive  behaviors  in  the  fictitious 
profiles.  This  has  some  basis  in  reality  because  raters  rarely  use  any  type  of  negative 
behavior  when  they  formally  rate  junior  officers,  but  raters  can  and  do  consider  negative 
behaviors  in  their  decisions.  This  study  was  limited  to  positive  behaviors  because  one  of 
the  reasons  for  accomplishing  this  research  was  to  validate  or  invalidate  Hurry’s  findings 
(1995).  So,  this  effort  was  forced  to  use  the  behaviors  chosen  by  Hurry  so  there  would  be 
a  direct  link  to  her  research.  The  analysis  results,  however,  suggest  that  the  dimensional 
constructs  were  not  completely  captured  with  these  behavior  sets. 

Another  criticism  of  this  method  is  the  use  of  only  positive  behaviors.  This 
resulted  in  a  left  skewed  distribution  for  all  four  dimensions.  This  method  gave  the  data 
less  variance  than  otherwise  could  have  been  achieved  by  using  a  full  range  of  both 
positive  and  negative  behaviors.  This  may  have  identified  a  relationship  that  would  not 
hold  true  if  the  full  range  of  the  dimensional  constructs  were  considered. 

The  final  weakness  I  will  discuss  is  the  length  of  the  survey.  Most  respondents 
indicated  that  it  took  them  approximately  30  minutes  to  complete  the  survey,  and  many 
comments  indicated  that  the  survey  was  too  long.  The  nature  of  policy  capturing  with 
profiles  necessitates  a  long  survey,  however.  The  number  of  behavior  dimensions  drives 
the  number  of  behaviors  rated  in  each  profile.  The  rule-of-thumb  for  regressions  is  there 
should  be  10  items  for  every  variable  measured.  In  this  case,  there  were  four  dimensions 
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plus  one  overall  performance  variable,  necessitating  50  profiles.  This  effort  used  48 
profiles  because  it  is  close  to  50,  and  the  lists  of  behaviors  lent  themselves  to  48  profiles. 

Future  Research 

Future  research  in  this  area  should  expand  the  selected  sample  frame  so  that  the 
research  is  more  generalizable.  Further  research  could  be  targeted  at  the  population  of 
Air  Force  officers  or  in  different  organizations  to  identify  organizational  difference  in 
rater  priorities.  Any  further  research  using  this  method  should  also  consider  using 
behavior  sets  that  capture  the  full  range  of  behaviors  from  positive  to  negative. 

Another  important  addition  to  this  research  would  be  the  introduction  of  other 
personality  facets.  This  study  may  not  have  captured  the  correct  personality  facets  for 
predicting  rater  leniency  or  predicting  the  relative  importance  of  the  beta  weights.  To 
keep  the  survey  short  enough  to  have  a  meaningful  response  rate,  only  a  few  personality 
facets  can  be  examined  in  each  study,  but  there  is  room  for  theory  development  and 
testing  in  this  area. 

Further  research  could  also  be  done  to  explore  the  contrary  rater  leniency  results 
of  this  research  compared  to  the  rater  leniency  results  found  by  Bemardin,  Cooke,  and 
Villanova  (2000).  This  could  provide  valuable  insight  into  the  rating  leniency  debate  and 
provide  evidence  of  a  motivation  model  (Murphy  and  Cleveland,  1999)  versus  some 
other  type  of  model.  One  possibility  would  be  for  a  researcher  to  set  up  a  study  where 
raters  evaluated  fictitious  profiles  while  another  like  group  of  raters  evaluated  real  ratees. 
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The  results  suggest  that  this  method  for  capturing  policy  may  not  be  adequate. 
Hobson  and  Gibson  (1983)  suggest  that  a  more  accurate  method  of  capturing  rating 
policies  would  be  to  use  field  research  using  real  raters  and  ratees.  These  results 
highlight  the  difficulties  in  using  this  particular  method  and  add  some  weight  to  Hobson 
and  Gibson’s  assertion.  The  results  tended  to  become  less  amenable  to  analysis  as  the 
research  moved  further  away  from  the  raw  data.  In  fact,  the  second  level  regression 
poses  a  high  level  of  difficulty  because  of  the  restraint  imposed  by  using  only  one 
dependent  variable.  The  beta  weights  lose  their  meaning  when  used  individually  because 
their  original  meaning  came  from  the  fact  that  they  are  relative  to  each  other.  That 
relativity  disappears  when  each  dimension  is  treated  separately  as  a  dependent  variable. 
This  method  may  be  able  to  find  the  policy,  but  it  may  never  be  able  to  answer  the 
question,  “Why  do  these  people  have  this  policy?”  Also,  a  survey  instrument  may  be 
inadequate  to  capture  the  full  range  of  the  dimensional  constructs  and  thus  an  accurate 
representation  of  rater’s  policies  unless  it  was  so  lengthy  that  no  respondents  would 
complete  it. 

A  field  study  would  certainly  be  more  time  consuming  and  difficult  to  carry  out 
than  the  method  used  in  this  study,  but  the  results  may  be  more  accurate  because  a  full 
range  of  behaviors  could  be  observed.  Also,  the  raters’  decisions  could  be  discussed  to 
gain  more  insight  into  their  method  for  using  various  performance  dimensions  to  arrive  at 
an  overall  performance  rating. 
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Appendix  A.  Leadership  Behaviors 


Table  30.  (Modified  from  Hurry,  1995:  16) — Leadership  Behaviors 


Behavior 

Reference 

Means 

Represents  the  group  effectively 

1 

5.49 

Assigns  subordinates  duties  and  responsibilities  appropriate  for  their  abilities 

1,5,6 

5.28 

Works  to  create  an  effective  unit  atmosphere 

3,6 

5.82 

Monitors  the  status  of  work  in  progress 

2 

5.59 

Reacts  confidently  when  the  unexpected  occurs 

2,3 

5.97 

Recognizes  and  encourages  effective  performance 

1,4,6 

5.76 

Ensures  deadlines  and  performance  standards  are  met 

5 

5.72 

Resolves  conflicts  between  members  of  the  unit 

4,6 

5.50 

Keeps  subordinates  focused  on  mission  requirements 

7 

5.55 

Resolves  conflicting  organizational  demands 

4,6 

5.31 

Maintains  high  visibility  both  on  and  off  the  job 

4 

4.00 

Guides  and  directs  subordinates  effectively 

5 

5.53 

Uses  good  judgment  in  decision  making 

2 

5.08 

Coordinates  subordinates'  efforts  to  minimize  conflicts 

2,4,6 

5.23 

Behaves  consistently  with  subordinates 

1,4,6 

5.12 

Takes  a  position  on  controversial  issues 

1,3 

5.46 

Persuades  others  both  inside  and  outside  the  organization 

3,6 

4.68 

Supports  subordinates 

1,2, 3, 4, 5,6 

4.74 

Avoids  trespassing  on  others'  responsibility  areas 

4 

6.14 

Speaks  effectively 

4 

6.01 

Makes  tough  decisions  quickly 

2 

5.69 

Provides  appropriate  feedback  to  subordinates 

1,2, 3, 4, 5,6 

5.35 

Sources:  IBausum  (1986),  2Borman  and  Motowidlo  (1993),  3  Air  Command  and  Staff  College 
(1988),  4Conger  et  al.  (1989),  5Van  Scotter  and  Shane  (1995),  6Borman  and  Brush  (1993),  and 
7Hurry  (1995). 
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Appendix  B.  Task  Performance  Behaviors 


Table  31.  (Modified  from  Hurry  1995:  18) — Task  Performance  Behaviors 


Behavior 

Reference 

Means 

Anticipates  potential  problems 

2,3 

5.04 

Operates  equipment  skillfully 

1 

5.77 

Uses  technical  material  effectively 

1,2,3 

5.68 

Communicates  task  information  effectively 

1,3 

5.35 

Performs  routine  tasks  efficiently 

1,2,3 

4.97 

Writes  clearly  and  concisely 

1,3 

4.99 

Keeps  up  with  the  newest  technology 

1,2,3 

5.86 

Performs  specialized  tasks  skillfully 

1,2,3 

5.66 

Performs  safely 

1,2 

5.93 

Solves  urgent,  unexpected  problems  expertly 

2 

4.14 

Uses  equipment,  tools,  and  computers  proficiently 

1,3 

5.70 

Provides  others  with  current  technical  information 

2,3 

5.92 

Plans  and  organizes  work 

1,2,3 

5.03 

Accomplishes  job  tasks  expertly 

1,2,3 

6.04 

Prioritizes  work  tasks  efficiently 

1,2,3 

5.84 

Uses  technical  expertise  to  meet  real  world  needs 

1,2 

5.46 

Troubleshoots  expertly 

2,3 

5.57 

Solves  technical  problems  expertly 

1,2,3 

5.31 

Collects  and  accurately  interprets  information 

1,2,3 

5.61 

Provides  expert  technical  advice  to  others 

1,2,3 

5.45 

Sources:  ’Department  of  the  Air  Force  (1988),  2Van  Scotter  (1994:  20-24),  and  3Borman 
and  Brush  (1993)  _ 
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Appendix  C.  Interpersonal  Facilitation  Behaviors 


Table  32.  (Modified  from  Hurry  1995:  19) — Interpersonal  Facilitation  Behaviors 


Behavior 

Reference 

Means 

Acts  warm  and  sociable 

5 

5.88 

Voluntarily  pitches  in  to  help  the  group 

1,2 

3.86 

Treats  others  fairly 

2 

5.04 

Offers  to  help  others  do  their  work 

1,2,3 

5.53 

Says  things  to  reduce  conflicts 

5 

4.99 

Avoids  arguments 

5 

6.00 

Supports  or  encourages  coworkers 

1,2 

4.47 

Talks  to  others  before  taking  actions  that  affect  them 

1,2 

5.01 

Coordinates  actions  with  others 

1,4 

5.89 

Gives  co workers  advice  about  how  to  do  their  jobs 

1,2 

5.19 

Develops  and  maintains  good  working  relationships 

2,4 

4.41 

Acts  courteously 

5 

4.88 

Helps  someone  without  being  asked 

1,2,3 

5.20 

Gets  along  with  others 

5 

5.14 

Displays  a  cheerful,  confident  outlook 

2,3 

4.05 

Displays  concern  for  others 

3,4 

5.86 

Says  things  to  make  people  feel  good  about  themselves 

5 

5.35 

Encourages  others  to  overcome  differences  and  get  along 

5 

4.54 

Encourages  coworkers  to  stick  together  in  hard  times 

1 

6.12 

Shows  respect  for  others 

2,3,4 

5.84 

Offers  friendly  advice 

5 

5.88 

Cooperates  with  others  in  the  team  effectively 

1,2 

5.92 

Listens  to  others'  ideas  about  getting  work  done 

1,2 

5.12 

Praises  coworkers  when  they  are  successful 

2 

5.81 

Lends  a  hand  when  a  coworker  needs  it 

1,2,3 

6.11 

Seeks  others  opinions 

5 

4.89 

Sources:  'Borman  and  Motowidlo  (1983),  2Van  Scotter  (1994), 

Van  Scotter  and 

Shane  (1995),  4Borman  and  Brush  (1993)  and  'Hurry  (1995). 
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Appendix  D.  Job  Dedication  Behaviors 


Table  33.  (Modified  from  Hurry,  1995:  21) — Job  Dedication  Behaviors 


Behavior 

Reference 

Means 

Overcomes  obstacles  to  complete  a  task 

2,4. 

5.70 

Takes  responsibility  for  his/her  actions 

5 

5.76 

Tackles  a  difficult  work  assignment  enthusiastically 

1,2,4 

5.35 

Adapts  to  difficult  conditions 

2,4 

5.68 

Takes  the  initiative  to  solve  a  work  problem 

1,2 

5.51 

Puts  extra  effort  into  a  task 

1,2, 3, 4 

5.95 

Ensures  work  is  done  right 

2 

Shows  respect  for  authority 

2,4 

Gives  up  personal  time  for  the  mission 

2 

Performs  consistently  and  reliably 

2,3 

Overcomes  hardships 

1,2,4 

Displays  proper  military  appearance  and  bearing 

1,2,4 

Puts  in  extra  hours  to  get  work  done  on  time 

Follows  the  supervisor's  instructions 

5 

4.95 

Renders  proper  military  courtesy 

1,2,4 

5.39 

Avoids  shortcuts  when  work  is  overdue 

5 

5.62 

Asks  for  challenging  work  assignments 

5 

4.97 

Complies  with  instructions  even  when  supervision  is  not  present 

1,2 

5.97 

Strives  to  excel 

2,3 

5.89 

Works  hard 

2,3 

6.11 

Pays  close  attention  to  important  details 

1,2 

4.84 

Defends  the  supervisor's  decisions 

2 

5.58 

Volunteers  for  additional  duties 

1,2 

5.32 

Volunteers  for  difficult  assignments 

2 

4.92 

Sources:  ‘Borman  and  Motowidlo  (1983),  2Van  Scotter  (1994),  3Van  Scotter  and  Shane 
(1995),  4Borman  and  Brush  (1993),  and  5Hurry  (1995). _ 
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Appendix  E.  Beta  Weights 


Table  34.  Beta  Weights  and  R  Values  (n=48  for  every  regression) 


Respondent 

LD 

TP 

IF 

JD 

R2 

Respondent 

LD 

TP 

IF 

JD 

R2 

1 

0.401 

0.185 

0.245 

0.166 

0.759 

37 

0.368 

0.299 

0.262 

0.234 

0.832 

2 

mm 

0.216 

mm 

■BE 

0.215 

■aM 

3 

mm 

0.335 

mm 

mm 

K 

0.228 

mm 

| 

4 

0.338 

0.213 

0.23 

0.296 

0.72 

0.321 

■ 

5 

0.306 

0.272 

0.26 

0.237 

0.502 

41 

Mill 

0.268 

BUT 

| 

6 

mm 

■mu 

mSEzL 

BaMfe 

7 

■urai 

0.325 

B*fe 

Bill 

0.18 

0.283 

0.214 

0.621 

8 

0.317 

Bum 

BMEK 

MIME 

9 

BflSH: 

0.252 

1 

45 

0.287 

HSB&] 

10 

0.346 

BiKEEl 

Bill 

46 

0.47 

■ 

■DEES 

11 

0.227 

0.13 

0.465 

0.297 

0.704 

47 

0.138 

0.291 

0.361 

BEH 

MME 

12 

0.342 

0.042 

0.419 

0.15 

0.611 

48 

0.161 

0.068 

0.087 

BHii 

BH8EI 

13 

0.206 

0.279 

0.065 

0.407 

0.822 

49 

0.235 

0.207 

0.065 

0.257 

mm 

14 

mm 

mm 

0.321 

mm 

0.439 

0.187 

0.245 

0.73 

15 

mm 

KBi 

0.059 

0.184 

51 

mm 

0.3 

0.224 

0.876 

16 

mm 

mm 

mm 

52 

mm 

BiUkv. 

BBS 

17 

mm 

mm 

mm 

53 

mm 

0.874 

18 

0.219 

-0.219 

0.353 

-0.005 

0.881 

54 

mme 

BKHB 

0.777 

19 

0.282 

0.239 

0.013 

0.169 

0.428 

55 

0.718 

20 

mm 

mm 

Bill 

56 

BiWife! 

0.657 

21 

57 

mm 

0.568 

22 

BBW 

58 

wm 

■ 

0.841 

23 

0.322 

0.256 

mm 

ebi 

EH 

59 

0.281 

0.235 

0.373 

0.264 

0.721 

24 

0.276 

0.26 

mm 

mm 

0.478 

0.233 

0.444 

0.298 

0.862 

25 

0.378 

0.291 

0.204 

0.246 

0.873 

61 

0.369 

BRIM 

wm 

26 

0.121 

0.122 

0.152 

0.242 

0.395 

62 

0.296 

MB 

mm 

27 

0.25 

0.283 

0.393 

0.428 

0.576 

63 

■Utirlfl 

IBilil 

MM 

28 

0.272 

0.24 

0.25 

0.218 

0.879 

MKEMM 

mm 

29 

0.239 

0.304 

0.196 

0.303 

0.872 

| 

lai 

mm 

EB 

■m 

30 

0.223 

0.218 

0.263 

0.28 

0.874 

66 

0.188 

0.293 

0.161 

0.178 

0.744 

31 

0.294 

0.184 

0.381 

0.27 

0.834 

67 

0.189 

0.368 

0.181 

0.265 

0.583 

32 

0.366 

0.199 

0.281 

0.198 

0.742 

68 

0.285 

0.203 

0.139 

0.323 

0.741 

33 

0.319 

0.36 

0.333 

0.352 

0.786 

69 

0.228 

0.289 

0.265 

0.219 

0.892 

34 

0.245 

0.202 

0.297 

0.102 

0.745 

1SII 

0.263 

mm 

wm\ 

35 

0.164 

0.247 

0.237 

0.222 

0.863 

71 

0.135 

0.209 

0.269 

0.287 

0.651 

36 

0.308 

0.19 

0.395 

0.203 

0.832 

72 

0.38 

0.285 

0.361 

0.423 

0.761 

75 


Table  34.  Continued 


83 

84 

85 

86 

87 

Respondent  LD 

73  0.338 

74  0.297 

75  0.215 

76  0.307 

77  0.295 


0.156 

0.341 


0.243 


81  0.323 

82  0.261 


83  0.235 

84  0.307 

85  0.208 
0.308 
0.157' 

88  1  0.611 

89  0.309 

90  0.265 ' 

91  0.263 

92  0.195 

93  j  026 
■301' 
.192' 


0.281 


TP 

0.169 

0.222’ 


0.382 

0.185 

0.3l" 


0.317 
0.243 " 


0.203 


0.354 
0.26  ~ 


0.235 
0.217’ 
0.289  ~ 
0.35 
0.236 
-0.056 
0.261 
0.233 
0.215 
0.294 

_ 0. 171 

_ 0 

0 


0.292 


IF 

0.261 ' 
0.149~ 
0.113' 
0.259' 
0.246’ 


0.219 

0.193’ 


0.205 


0.178 

0.278’ 


0.103 
0.23 1~ 
0.265’ 


0.279 

0.338’ 


0.278 


0.198 
0.255 " 


0.212 
0.246" 
0.084  ’ 


,11) 

0.37' 

0.123 

0.212' 

0.274' 

0.29l’ 


0.179 

0.27’ 


0.395 


0.418 

0.262" 


0.276 
0.266’ 
0.212~ 
0.145 
0.182~ 
0.167’ 
0.287~ 
0.289  ~ 
0.248" 
0.254" 
0464’ 
.247 
.218 
0.056 


R2  Respondent 
0.64  111 

0.442 _ 112 

0.529 _ 

0.814 _ 

0.925 


0.713 

0.674 


0.844 


0.711  119 

0.586  120 


0.661 
0.663 

0.768 _ 

0.733 _ 

0.709 _ 

0.905 _ 126 

0.86 _ 

_ 041 _ 

0.67 _ 

0.806 _ 

0.417 _ 

0.864 _ 

0.748 _ 

0.59 


LD  TP  IF  JD 

0.309  0.355  0.234  0.281~ 

0.391  0.1471  04861  049’ 

.226 


0.18  0.209 


Table  34.  Continued 


Respondent 

LD 

TP 

IF 

JD 

R2 

Respondent 

LD 

TP 

IF 

JD 

R2 

149 

0.346 

0.366 

0.272 

0.441 

0.767 

190 

BHil 

0.211 

mm 

KM&! 

mmm 

0.291 

0.327 

0.351 

0 

0.621 

191 

BPK5 

■m 

151 

0.289 

0.269 

0.359 

0.22 

0.847 

192 

Blfe 

m m 

KHE 

152 

0.287 

0.354 

0.177 

0.227 

0.649 

193 

0.813 

0.259 

0.213 

0.223 

KPS 

153 

0.485 

0.266 

0.396 

0 

0.406 

194 

0.301 

0.262 

0.109 

0.272 

B 

154 

0.371 

0.295 

0.216 

0.209 

0.907 

195 

Must: 

beui 

Bing 

KM 

155 

0.296 

0.254 

0.257 

0.231 

0.828 

196 

BEKS 

Mur/.v 

156 

0.363 

0.28 

0.123 

0.318 

0.845 

197 

EH 

BBB 

MMjti 

157 

0.333 

0 

0.599 

0.481 

0.765 

198 

■ 

mm\ 

158 

0.276 

0.364 

0.04 

0.349 

0.856 

199 

Bums 

UUbHIKj 

159 

0.29 

0.255 

0.201 

0.303 

0.809 

200 

■ 

B0H! 

0.453 

160 

0.19 

0.275 

0.0699 

0.146 

0.44 

201 

1 

0.784 

161 

0.348 

0.247 

0.284 

0.308 

0.656 

202 

KIM 

mm 

0.633 

162 

0.347 

0.235 

0.259 

0.324 

0.819 

203 

BKM 

BBSS 

0.866 

163 

0.298 

0.268 

0.347 

0.175 

0.641 

204 

BBSSK 

B^tin 

164 

0.306 

0.222 

0.234 

0.286 

0.725 

205 

KgJE 

BKTO 

KKS 

165 

0.45 

0.274 

0.206 

0.223 

0.739 

kto 

mm 

166 

0.23 

0.256 

mm 

mm 

Mam 

K&B! 

mm 

167 

0.326 

0.299 

mm 

mm 

mm 

KISH 

IBM 

mm 

168 

0.267 

0.283 

0.293 

0.287 

0.912 

209 

BBM 

KM 

KM! 

mm 

169 

0.2 

0.185 

0.251 

0.289 

0.798 

210 

mm 

170 

0.327 

0.181 

0.238 

0.258 

0.824 

211 

BblM 

mm 

171 

0.337 

0.165 

0.255 

0.311 

0.889 

212 

iftiKII 

mm 

mm 

172 

0.379 

0.156 

0.346 

0.096 

0.65 

213 

Bllfi 

5H 

m m 

mm 

173 

0.372 

0.384 

0.341 

0.215 

0.753 

214 

1 

BRUSH 

0.218 

0.61 

174 

0.315 

0.208 

0.184 

0.166 

0.806 

215 

HH! 

BM 

175 

0.354 

0.224 

0.217 

0.187 

0.694 

216 

mm 

EH 

KMl 

176 

0.049 

0.093 

0.126 

0.035 

0.216 

217 

0.363 

0.197 

0 

0.35 

0.718 

177 

0.437 

0.301 

0.115 

0.288 

0.869 

218 

0.4 

0.23 

0.309 

0 

0.559 

178 

0.175 

0.312 

0.291 

0.321 

0.813 

219 

0.269 

0.276 

0.178 

0.143 

0.811 

179 

0.309 

0.213 

0.179 

0.268 

0.776 

220 

0.313 

0.083 

0.159 

0.377 

0.599 

180 

0.28 

0.075 

0.272 

0.301 

0.868 

221 

0.234 

0.317 

0.212 

0.195 

0.591 

181 

0.274 

0.157 

0.304 

0.226 

0.671 

222 

0.21 

0.27 

0.234 

0.225 

0.685 

182 

0.236 

0.198 

0.343 

0.307 

0.855 

223 

0.312 

0.211 

0.162 

0.215 

0.685 

183 

0.223 

0.23 

0.261 

0.258 

0.62 

224 

0.149 

0.153 

0.393 

0.307 

0.764 

184 

0.172 

0.235 

0.243 

0.254 

0.659 

225 

0.52 

0.304 

0.119 

0.375 

0.584 

185 

0.003 

0.464 

0.228 

0.333 

0.68 

226 

0.209 

0.2 

0.121 

0.054 

0.257 

186 

0.329 

0.345 

0.169 

0.202 

0.684 

227 

0.1 

0.454 

0.063 

0.294 

0.82 

187 

0.29 

0.271 

0.155 

0.315 

0.82 

228 

0.146 

0.246 

0.276 

0.276 

0.83 

188 

0.236 

0.324 

0.309 

0.249 

0.701 

229 

0.262 

0.313 

0.236 

0.228 

0.619 

|  189 

0.426 

0.292 

0.254 

0.398 

0.587 
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Appendix  F.  Survey 


DETERMINING  IMPORTANT 
BEHAVIORS 
FOR 

JUNIOR  OFFICER  SUCCESS 

ABOUT  THIS  SURVEY 

Purpose:  The  purpose  of  this  survey  is  to  assess  the  relative  importance  of  four  types  of  behaviors  displayed  by 
company  grade  officers—job  dedication,  interpersonal  facilitation,  leadership,  and  task  performance.  Your 
participation  will  help  us  determine  the  types  of  behaviors  that  people  in  the  Air  Force  consider  most  important  for 
junior  officers  to  possess. 

Content:  The  survey  is  divided  into  three  parts.  The  first  part  contains  some  demographic  questions  to  make 
comparisons  between  groups.  The  second  section  contains  behavior  profiles  for  some  fictitious  company  grade  officers. 
Each  profile  consists  of  the  name  of  the  individual,  four  behaviors,  the  category  of  each  behavior,  scales  for  rating  each 
of  the  four  behaviors,  and  a  scale  for  rating  the  overall  performance  of  the  individual.  The  final  section  contains  a 
series  of  questions  that  measure  personality  traits.  These  will  be  used  to  assess  differences  across  all  people  who 
participate  in  this  survey.  It  should  take  approximately  40  minutes  to  complete  this  questionnaire. 

Confidentiality:  Your  answers  are  STRICTLY  CONFIDENTIAL,  and  will  be  anonymous  unless  you  wish  to 
disclose  your  identity.  Findings  will  be  reported  at  the  group  level  only,  so  no  one  will  be  able  to  trace  your  responses 
back  to  you.  No  one  outside  my  research  team  will  see  your  questionnaire. 

When  Complete:  Please  place  the  completed  survey  in  the  return  envelope  provided  and  place  it  in  the  standard  Air 
Force  mail  distribution  system.  We  appreciate  your  response  -  it  is  important  and  essential  to  the  success  of  the  study. 

Contact:  If  you  have  questions  or  comments  about  the  survey,  please  contact  me  or  my  advisor  by  telephone  or 
through  electronic  mail.  We  sincerely  thank  you  for  your  participation. 


Lt.  Owen  Stephens 
owen.s1ephens@aiit.af.mil  or 
us3stephens@earthlink.net 
(937)667-5184 


Maj.  Michael  Rehg 
michael.rehg@afit.af.mil 
Commercial:  (937)  255-3636  x471 1 
DSN:  785-3636x4711 


USAF  Survey  Control  Number  00-94 
Expires  31  Dec  00 
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Parti.  Information  About  You 


1.  What  is  your  rank? 

□  2nd  Lt  □  ls,Lt  □  Capt  □  Major  □  LtCol  □  Col 

2.  Please  indicate  your  gender: 

□  Male  □  Female 

3.  Please  indicate  the  number  of  total  active  military  service  years  you  have: 

_ Years 

4.  If  prior  enlisted,  how  many  of  your  total  active  military  service  years  were  spent  as 
enlisted? 


Years 


5.  What  is  your  primary  job  category?  (If  you  are  not  currently  in  one  of  these  categories, 
please  choose  the  one  that  best  describes  the  major  portion  of  your  career  or  use  the 
"other"  option.) 

□  Rated  Officer  (pilot,  navigator,  etc.) 

□  Support  Officer  (logistician,  personnel,  security  police,  finance,  etc.) 

CD  Analysis/Engineer 

CD  Other  (please  specify) _ 


Supervisor  is  defined  here  as  direct  reporting  official.  To  qualify  as  a 
supervisor, 

you  must  have  had  to  rate  an  individual  on  their  performance. 

6-  Approximately  how  many  years  of  experience  do  you  have  as  a  supervisor? 

□  0  □  1-4  □  5-8  □  9-13  □  14  or  more 

7.  Approximately  how  many  officers  have  you  rated  in  your  career? 

□  0  □  1-3  □  4-6  □  7-10  □  11  or  more 

8.  What  is  your  source  of  commission? 

□  USAF  Academy  □  OTS 

D  ROTC  □  Other  (please  specify) _ 


Part  II.  The  Profiles 


In  this  portion  of  the  survey  you  are  asked  to  rate  behaviors  of  fictitious  officers.  The  behaviors  represent  four  broad 
categories  of  behaviors  as  follows: 

Leadership — The  art  of  influencing  and  directing  people  to  accomplish  the  mission. 

Task  Performance — The  proficiency  with  which  an  individual  performs  activities  that  are  formally  recognized  as  the 
technical  or  specialized  activities  that  define  his  or  her  job. 

Interpersonal  Facilitation-— The  extent  to  which  a  worker  supports  other  members  of  the  organization  through 
expressions  of  concern,  consideration,  cooperative  and  helpful  acts. 

Job  Dedication— The  extent  to  which  a  worker  goes  above  and  beyond  the  requirements  of  his  or  her  official  job 
description  and  exceeds  the  minimum  expectations  of  his  or  her  rater. 
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Read  the  behaviors  within  each  profile.  Then,  rate  each  behavior  and  the  overall  performance  of  the  officer  by  placing 
an  X  in  the  box  that  you  feel  best  describes  the  performance  of  each  officer  accordingly. 

Example: 


Joe  Smith 


Behavior  Type 


Examples  of  “On  the  Job"  Behaviors 


Low 

High 


Behavior  Rating 
Medium 


Leadership  Leads  effortlessly 

Job  Dedication  Works  really  hard  most  of  the  time 

Interpersonal 

Facilitation  Rarely  fights  with  others 

Read  these  example 
behaviors,  then  place  an  X  in 
one  of  the  boxes  here  for 
each  behavior. 

_ k  □□□□□ED 

))  □□□□□□  El 

Overall  Rating 

Low 


Medium 


High 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overal I  performance 


Officer  Profiles 


□  □□□□an 


IDT 


Then  rate  the 
overall  performance  of  the 
officer  by  placing  an  X  in 
one  of  these  boxes 


Bill  Cox 


Behavior  Rating 
Low  Medium 

HI _ 

□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 

□  □  □  □  □  n  n 


Overall  Rating 
Low  Medium 

_ _ _ _ _ _ _  High _ 

Please  mark  the  box  E  that  best  reflects  this  subordinates  overall  performance  ± _ □□□□□□□ 

Gary  Fuller  ' 


Behavior  Type _ Examples  of  “On  the  Job"  Behaviors 

Task  Performance  Anticipates  potential  problems 

Leadership  Represents  the  group  effectively 

Job  Dedication  Overcomes  obstacles  to  complete  a  task 

Interpersonal 

Facilitation _ Acts  warm  and  sociable 


Behavior  Type 

Examples  of  “On  the  Job”  Behaviors 

Low 

High 

Behavior  Rating 
Medium 

Interpersonal 

Facilitation 

Leadership 

Job  Dedication 

Task  Performance 

Voluntarily  pitches  in  to  help  the  group 

Recognizes  and  encourages  effective  performance 

Takes  responsibility  for  his/her  actions 

Operates  equipment  skillfully 

□ 

□ 

□ 

□ 

□□□□ 

□□□□ 

□□□□ 

□□□□ 

□□□□ 

□ 

□ 

□ 

□ 

Overall  Rating 

Low  Medium 

High 

Please  mark  the  box  1X1  that  best  reflects  this  subordinate’s  overall  performance 

□ 

□ 

□ 

□ 

□ 

□ 

□ 
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Mark  Farrior 


Behavior  Type 


Examples  of  “On  the  Job”  Behaviors 


Low 

High 


Behavior  Rating 
Medium 


Task  Performance 
Interpersonal 
Facilitation 
Job  Dedication 
Leadership 


Uses  technical  material  effectively 
Treats  others  fairly 

Tackles  a  difficult  work  assignment  enthusiastically 
Takes  a  position  on  controversial  issues _ 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □□□□□□ 


Overall  Rating 
Low  Medium 

_ _ _  High _ 

Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance  _ D  D  D  D  D  D  □ 

Elaine  Simpson 


Behavior  Type 

Examples  of  “On  the  Job”  Behaviors 

Low 

High 

Behavior  Rating 
Medium 

Interpersonal 

Facilitation 

Avoids  arguments 

□ 

□  □  □  □  □ 

□ 

Task  Performance 

Performs  safely 

□ 

□  □  □  □  □ 

□ 

Leadership 

Makes  tough  decisions  quickly 

□ 

□  □  □  □  □ 

□ 

Job  Dedication 

Renders  proper  military  courtesy 

□ 

□  □  □  □  □ 

□ 

Overall  Rating 
Low  Medium 

_ _ _  High _ 

Please  mark  the  box  ESI  that  best  reflects  this  subordinate's  overall  performance  -> _ □□□□□□□ 

Earl  Phillips 


Behavior  Type 
Interpersonal 
Facilitation 
Leadership 
Job  Dedication 
Task  Performance 


Examples  of  “On  the  Job”  Behaviors 


Behavior  Rating 
Low  Medium 

High _ 


Acts  warm  and  sociable  □□□□□□□ 

Speaks  effectively  □□□□□□□ 

Follows  the  supervisor's  instructions  □□□□□□□ 

Uses  technical  material  effectively _  □□□□□□□ 


Overall  Rating 
Low  Medium 

_ _ _ High _ 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  -> _ □□□□□□□ 


Brian  Anderson 


Behavior  Type 

Examples  of  “On  the  Job”  Behaviors 

Low 

High 

Behavior  Rating 
Medium 

Interpersonal 

Facilitation 

Avoids  arguments 

□ 

□  □  □  □  □ 

□ 

Leadership 

Maintains  high  visibility  both  on  and  off  the  job 

□ 

□  □  □  □  □ 

□ 

Job  Dedication 

Puts  extra  effort  into  a  task 

□ 

□  □  □  □  □ 

□ 

Task  Performance 

Writes  clearly  and  concisely 

□ 

□  □  □  □  □ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ High _ 

Please  mark  the  box  E3  that  best  reflects  this  subordinate's  overall  performance  _ □□□□□□□ 
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Harold  Echard 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ High _ 


Leadership 

Monitors  the  status  of  work  in  progress 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Keeps  up  with  the  newest  technology 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Supports  or  encourages  coworkers 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Ensures  work  is  done  right 

□ 

□ 

n 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ HiSh _ 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance  _ □□□□□□□ 

Chuck  Porter  — — —  ■  ;  . 


Behavior  Rating 
Low  Medium 

Behavior  Type  Examples  of  uOn  the  Job”  Behaviors _ High _ 


Interpersonal 

Facilitation 

Cooperates  with  others  in  the  team  effectively 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Provides  others  with  current  technical  information 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Gives  up  personal  time  for  the  mission 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Persuades  others  both  inside  and  outside  the  organization 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ High _ 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  _ □□□□□□□ 

Susan  Gates  ~ 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ High _ 


Leadership 

Coordinates  subordinates'  efforts  to  minimize  conflicts 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Performs  specialized  tasks  skillfully 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Facilitation 

Talks  to  others  before  taking  actions  that  affect  them 

Job  Dedication 

Shows  respect  for  authority 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ High _ 

Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance  _ □□□□□□□ 


Bill  Taylor 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ High _ 


Interpersonal 

Facilitation 

Coordinates  actions  with  others 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Works  to  create  an  effective  unit  atmosphere 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Plans  and  organizes  work 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Volunteers  for  difficult  assignments 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Please  mark  the  box  ^  that  best  reflects  this  subordinate's  overall  performance 


Overall  Rating 
Low  Medium 

High _ 

□  □□□□□□ 
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Amy  Godwin 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Task  Performance  Solves  urgent,  unexpected  problems  expertly 
Interpersonal 

Facilitation  Gives  coworkers  advice  about  how  to  do  their  jobs 

Job  Dedication  Performs  consistently  and  reliably 

Leadership _ Avoids  trespassing  on  others ‘  responsibility  areas 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  -> 

Charles  Zimmer  — 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 
□  □  n  n  n  n  n 


Overall  Rating 
Low  Medium 

High 

□  □  □  □  □  n  n 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Interpersonal 

Facilitation  Acts  courteously 

Job  Dedication  Puts  in  extra  hours  to  get  work  done  on  time 

Task  Performance  Prioritizes  work  tasks  efficiently 
Leadership _ Provides  appropriate  feedback  to  subordinates 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance 

Kevin  Meadows 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 
□□□□□□□ 
□□□□□□□ 
□  □□□□□  □ 


Overall  Rating 
Low  Medium 

High 

□  □□□□□□ 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Adapts  to  difficult  conditions 

Leadership  Guides  and  directs  subordinates  effectively 

Task  Performance  Solves  technical  problems  expertly 
Interpersonal 

Facilitation _ Develops  and  maintains  good  working  relationships 


Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance 

Hank  Marti n  ~  ~~  .  ~ 


Behavior  Rating 
Low  Medium 

High _ _____ 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Leadership  Uses  good  judgment  in  decision  making 

Interpersonal 

Facilitation  Displays  a  cheerful,  confident  outlook 

Job  Dedication  Defends  the  supervisor's  decisions 

Task  Performance  Uses  equipment,  tools,  and  computers  proficiently 


Please  mark  the  box  [El  that  best  reflects  this  subordinate’s  overall  performance  -> 


Behavior  Rating 
Low  Medium 

High 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 
□  □  □  □  □  n  n 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□  n 
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Andrew  Harris 


Behavior  Type 
Leadership 
Job  Dedication 
Task  Performance 
Interpersonal 
Facilitation 


Examples  of  “On  the  Job”  Behaviors _ 

Keeps  subordinates  focused  on  mission  requirements 
Adapts  to  difficult  conditions 
Communicates  task  information  effectively 

Offers  to  help  others  do  their  work 


Behavior  Rating 
Low  Medium 

High _ 

□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


Please  mark  the  box  0  that  best  reflects  this  subordinate’s  overall  performance 


Overall  Rating 
Low  Medium 

High 

□  □□□□□□ 


Gregg  Young 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ High _ 


Task  Performance 

Anticipates  potential  problems 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Displays  proper  military  appearance  and  bearing 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Behaves  consistently  with  subordinates 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Helps  someone  without  being  asked 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ _ High _ 

Please  mark  the  box  E]  that  best  reflects  this  subordinate's  overall  performance  _ □□□□□□□ 

John  Adams  — —  _ 


Behavior  Type 


Examples  of  “On  the  Job”  Behaviors 


Behavior  Rating 
Low  Medium 


Task  Performance 

Accomplishes  job  tasks  expertly 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Displays  concern  for  others 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Resolves  conflicts  between  members  of  the  unit 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Avoids  shortcuts  when  work  is  overdue 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _  High _ 

Please  mark  the  box  ^  that  best  reflects  this  subordinate's  overall  performance  -> _ D  D  D  D  □  □  □ 

Pamela  Davis  ~~~ 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ High _ 


Leadership 

Supports  subordinates 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Says  things  to  make  people  feel  good  about  themselves 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Asks  for  challenging  work  assignments 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Prioritizes  work  tasks  efficiently 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _  High 

Please  mark  the  box  [><]  that  best  reflects  this  subordinate’s  overall  performance  -> _ □□□□□□□ 

Larry  Lane 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _  High _ 


Task  Performance 

Writes  clearly  and  concisely 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Reacts  confidently  when  the  unexpected  occurs 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Asks  for  challenging  work  assignments 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Encourages  others  to  overcome  differences  and  get  along 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance 


Overall  Rating 
Low  Medium 

High 

□  □□□□□□ 


85 


Alan  Larson 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Leadership  Avoids  trespassing  on  others'  responsibility  areas 

Task  Performance  Solves  urgent ,  unexpected  problems  expertly 
Interpersonal 

Facilitation  Offers  to  help  others  do  their  work 

Job  Dedication _ Overcomes  obstacles  to  complete  a  task _ 


Please  mark  the  box  [El  that  best  reflects  this  subordinate’s  overall  performance  -> 

Lance  Kennedy 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□  n 


Behavior  Type 
Leadership 
Interpersonal 
Facilitation 
Job  Dedication 
Task  Performance 


Examples  of  “On  the  Job”  Behaviors _ 

Resolves  conflicting  organizational  demands 

Develops  and  maintains  good  working  relationships 
Overcomes  hardships 

Anticipates  potential  problems _ 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □  □  □  □  n  n 


Overall  Rating 
Low  Medium 

- _ _ _ _ _ High _ 

Please  mark  the  box  Kl  that  best  reflects  this  subordinate's  overall  performance  -> _ □  □  □  □  □  □  □ 

Judy  Vaughn  " 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors 

Interpersonal  ’  . 

Facilitation  Says  things  to  make  people  feel  good  about  themselves 

Task  Performance  Accomplishes  job  tasks  expertly 

Job  Dedication  Takes  responsibility  for  his/her  actions 

Leadership _ Recognizes  and  encourages  effective  performance 


Behavior  Rating 
Low  Medium 

High _ 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□  n 


Overall  Rating 
Low  Medium 

_ _ _  High 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  _ □  □  □  □  □  □  □ 

Ned  White  - - — — - 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Puts  extra  effort  into  a  task 

Interpersonal 

Facilitation  Listens  to  others'  ideas  about  getting  work  done 

Leadership  Maintains  high  visibility  both  on  and  off  the  job 

Task  Performance  Writes  clearly  and  concisely _ 


Behavior  Rating 
Low  Medium 

High _ _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 
□  □□□□□  n 


Overall  Rating 
Low  Medium 

_ _ _ _ __ _ High _ 

Please  mark  the  box  Kl  that  best  reflects  this  subordinate’s  overall  performance  _ □□□□□□□ 
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Thomas  Robinson 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Task  Performance  Collects  and  accurately  interprets  information 

Leadership  Makes  tough  decisions  quickly 

Job  Dedication  Ensures  work  is  done  right 

Interpersonal 

Facilitation _ Offers  friendly  advice _ 


Please  mark  the  box  IEI  that  best  reflects  this  subordinate’s  overall  performance 

Bruce  Hudson 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □  □  □  n  n  n 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Follows  the  supervisor's  instructions 

Interpersonal 

Facilitation  Gets  along  with  others 

Task  Performance  Provides  others  with  current  technical  information 
Leadership _ Provides  appropriate  feedback  to  subordinates 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance  -> 

Marcia  Moore  ~ 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □  □  □  □  n  n 


Overall  Rating 
Low  Medium 

High 

□  □□□□□  n 


Behavior  Type _ Examples  of  “On  the  Job"  Behaviors _ 

Leadership  Takes  a  position  on  controversial  issues 

Task  Performance  Communicates  task  information  effectively 
Interpersonal 

Facilitation  Encourages  coworkers  to  stick  together  in  hard  times 

Job  Dedication _ Volunteers  for  additional  duties 


Behavior  Rating 
Low  Medium 

High 

□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 


Overall  Rating 
Low  Medium 

- - - - High _ 

Please  mark  the  box  Ex]  that  best  reflects  this  subordinate’s  overall  performance  ^ _ □□□□□□  □ 

Patrick  Baker 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Works  hard 

Leadership  Avoids  trespassing  on  others’  responsibility  areas 

Interpersonal 

Facilitation  Offers  friendly  advice 

Task  Performance  Troubleshoots  expertly _ 


Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance 


Behavior  Rating 
Low  Medium 

High _ 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


□  □□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 
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Carolyn  Jackson 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job"  Behaviors _ High _ 

Leadership  Coordinates  subordinaies'  efforts  to  minimize  conflicts  □□□□□□□ 

Task  Performance  Troubleshoots  expertly  □□□□□□□ 

Job  Dedication  Pays  close  attention  to  important  details  □□□□□□□ 

Interpersonal  □□□□□□□ 

Facilitation _ Cooperates  with  others  in  the  team  effectively _ 


Overall  Rating 
Low  Medium 

_ _ _ _ _ High 

Please  mark  the  box  Ex]  that  best  reflects  this  subordinate’s  overall  performance  _ □  □  □  □  □  □  □ 

Robert  Long  “  ' 


Behavior  Type 
Job  Dedication 
Task  Performance 

Leadership 

Interpersonal 

Facilitation 


Examples  of  “On  the  Job”  Behaviors _ _ 

Shows  respect  for  authority  ~~ 

Provides  expert  technical  advice  to  others 

Assigns  subordinates  duties  and  responsibilities  appropriate  for  their 
abilities 

Gives  coworkers  advice  about  how  to  do  their  jobs _ 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □□□□□□ 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance 

Oscar  Olsen  ~~~ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors 

Task  Performance  Operates  equipment  skillfully 

Interpersonal 

Facilitation  Supports  or  encourages  a  coworker 

Leadership  Monitors  the  status  of  work  in  progress 

Job  Dedication _ Performs  consistently  and  reliably 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance 

Joseph  Evans  — — 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Takes  the  initiative  to  solve  a  work  problem 

Leadership  Supports  subordinates 

Task  Performance  Performs  routine  tasks  efficiently 

Interpersonal 

Facilitation _ Says  things  to  reduce  conflicts _ 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  -> 

gg 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □  □  □  n  n  n 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□  n 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


Overall  Rating 
Low  Medium 

High 

□  □□□□□□ 


Rachel  Jacobs 


Behavior  Type 
Interpersonal 
Facilitation 
Task  Performance 
Job  Dedication 
Leadership _ 


Examples  of  “On  the  Job”  Behaviors 


Behavior  Rating 
Low  Medium 

High _ 


Encourages  co  workers  to  stick  together  in  hard  times  □□□□□□□ 

Uses  technical  expertise  to  meet  real  world  needs  □□□□□□□ 

Complies  with  instructions  when  supervision  is  not  present  □□□□□□□ 

Uses  good  judgment  in  decision  making _ _ _ □□□□□□□ 


Overall  Rating 
Low  Medium 

_ _ _ _ _ _ _ _ High _ 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance  -> _ □□□□□□  □ 

Peter  Nelson  — 


Behavior  Type _ Examples  of  "On  the  Job”  Behaviors _ 

Interpersonal 

Facilitation  Shows  respect  for  others 

Leadership  Works  to  create  an  effective  unit  atmosphere 

Task  Performance  Keeps  up  with  the  newest  technology 
Job  Dedication _ Overcomes  hardships _ 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance 

Beth  Carpenter 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Overall  Rating 
Low  Medium 

High 

□  □□□□□  n 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Task  Performance  Uses  equipment,  tools ,  and  computers  proficiently 

Interpersonal 

Facilitation  Acts  courteously 

Job  Dedication  Displays  proper  military  appearance  and  bearing 

Leadership _ Persuades  others  both  inside  and  outside  the  organization 


Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance  -> 

Richard  Rivera  " 


Behavior  Rating 
Low  Medium 

High 

□  □□□□□□ 
□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type _  Examples  of  “On  the  Job”  Behaviors _ 

Job  Dedication  Pays  close  attention  to  important  details 

Task  Performance  Uses  equipment,  tools,  and  computers  proficiently 
Leadership  Resolves  conflicts  between  members  of  the  unit 

Interpersonal 

Facilitation _ Praises  coworkers  when  they  are  successful 


Behavior  Rating 
Low  Medium 

High 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


Overall  Rating 

Low  Medium 

High 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance 

□  □□□□□□ 
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Ronald  Slfford 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Leadership  Speaks  effectively 

Job  Dedication  Avoids  shortcuts  when  work  is  overdue 

Interpersonal 

Facilitation  Talks  to  others  before  taking  actions  that  affect  them 

Task  Performance  Uses  technical  expertise  to  meet  real  world  needs 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance 

David  Woods  ~  — ~  —  ~ 


Low 

High 


Behavior  Rating 
Medium 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


□  □  □  □  n  n  n 


Overall  Rating 
Low  Medium 

_ High _ 

□  □  □  □  n  n  n 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Leadership  Resolves  conflicting  organizational  demands 

Task  Performance  Provides  expert  technical  advice  to  others 

Job  Dedication  Works  hard 

Interpersonal 

Facilitation _ Treats  others  fairly _ 


Please  mark  the  box  IE1  that  best  reflects  this  subordinate’s  overall  performance  -> 

Brandon  Griffin  ” 


Behavior  Rating 
Low  Medium 

High 

□□□□□□□ 
□□□□□□□ 
□  □□□□□  n 
□□□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □  □  □  n  n  n 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ _ 

Interpersonal 

Facilitation  Praises  coworkers  when  they  are  successful 

Job  Dedication  Volunteers  for  additional  duties 

Task  Performance  Collects  and  accurately  interprets  information 
Leadership _ Reacts  confidently  when  the  unexpected  occurs 


Please  mark  the  box  K1  that  best  reflects  this  subordinate's  overall  performance  -> 

Bob  Jones  —  —  “ “ 


Behavior  Rating 
Low  Medium 

High _ 


□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _ 

Task  Performance  Accomplishes  job  tasks  expertly 
Interpersonal 

Facilitation  Shows  respect  for  others 

Leadership  Ensures  deadlines  and  performance  standards  are  met 

Job  Dedication  Sthves  to  excel 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate's  overall  performance  -> 


Behavior  Rating 
Low  Medium 

High _ 


□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 
□  □□□□□  n 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 
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Mark  Lorton 


Behavior  Type 
Interpersonal 
Facilitation 
Leadership 
Job  Dedication 
Task  Performance 


Examples  of  “On  the  Job”  Behaviors _ 

Lends  a  hand  when  a  coworker  needs  it 
Represents  the  group  effectively 
Tackles  a  difficult  work  assignment  enthusiastically 
Performs  routine  tasks  efficiently 


Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance 


Ethan  Duncan 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

Overall  Rating 
Low  Medium 

_ High _ 

□  □  □  □  □  □  □ 


Behavior  Type 
Task  Performance 
Leadership 
Interpersonal 
Facilitation 
Job  Dedication 


Examples  of  “On  the  Job”  Behaviors _ 

Solves  technical  problems  expertly 
Guides  and  directs  subordinates  effectively 

Listens  to  others '  ideas  about  getting  work  done 
Defends  the  supervisor's  decisions _ 


Behavior  Rating 
Low  Medium 

High 

□  □□□□□□ 
□  □□□□□□ 
□  □□□□□□ 

□  □□□□□□ 


Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance 

Nancy  Brewer 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type 
Interpersonal 
Facilitation 
Job  Dedication 
Leadership 
Task  Performance 


Examples  of  “On  the  Job”  Behaviors _ 

Seeks  others’  opinions 
Volunteers  for  a  difficult  assignment 
Behaves  consistently  with  subordinates 
Provides  expert  technical  advice  to  others 


Please  mark  the  box  [El  that  best  reflects  this  subordinate’s  overall  performance 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□□ 


Behavior  Type 
Job  Dedication 
Leadership 
Task  Performance 
Interpersonal 
Facilitation 


Examples  of  “On  the  Job”  Behaviors _ 

Renders  proper  military  courtesy 

Keeps  subordinates  focused  on  mission  requirements 

Plans  and  organizes  work 

Displays  a  cheerful  confident  outlook 


Behavior  Rating 
Low  Medium 

High _ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 

□□□□□□□ 


Please  mark  the  box  [x]  that  best  reflects  this  subordinate’s  overall  performance 


Overall  Rating 
Low  Medium 

_ High _ 

□  □□□□□  □ 
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Robert  Brown 


Behavior  Rating 
Low  Medium 

Behavior  Type _ Examples  of  “On  the  Job”  Behaviors _  High _ 


Task  Performance 

Performs  specialized  tasks  skillfully 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Makes  tough  decisions  quickly 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Puts  in  extra  hours  to  get  work  done  on  time 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Interpersonal 

Facilitation 

Helps  someone  without  being  asked 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ High _ 

Please  mark  the  box  E3  that  best  reflects  this  subordinate's  overall  performance  _ □□□□□□  □ 


Diana  Mendoza 


Behavior  Type 


Examples  of  “On  the  Job”  Behaviors 


Behavior  Rating 
Low  Medium 


Interpersonal 

Facilitation 

Says  things  to  reduce  conflicts 

-  ^ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Task  Performance 

Prioritizes  work  tasks  efficiently 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Leadership 

Reacts  confidently  when  the  unexpected  occurs 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Job  Dedication 

Strives  to  excel 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

Overall  Rating 
Low  Medium 

_ _ _ High 

Please  mark  the  box  [X]  that  best  reflects  this  subordinate’s  overall  performance  -> _ □□□□□□□ 


Part  111.  Personality  information] 

Please  use  the  rating  scale  below  to  describe  how  accurately  each  statement  applies  to  you.  Describe  yourself 
as  you  honestly  see  yourself,  in  relation  to  other  people  you  know  of  the  same  sex  as  you  are,  and  roughly  your  same 
age.  Please  read  each  statement  carefully,  and  then  place  an  X  in  the  square  that  BEST  reflects  the  extent  to  which  this 

behavior  describes  you. 


1 

2 

3 

4 

5 

Very  Inaccurate 

Inaccurate 

Neither  Inaccurate  nor 
Accurate 

Accurate 

Very  Accurate 

Like  to  act  on  a  whim. 

D 

B 

B 

B 

B 

Know  how  to  get  things  done. 

D 

b 

B 

D 

B 

Have  difficulty  starting  tasks. 

D 

B 

B 

B 

B 

Demand  quality. 

n 

B 

B 

B 

B 

Try  to  follow  the  rules. 

D 

H 

B 

B 

B 

Rush  into  things. 

n 

B 

B 

B 

B 

Am  always  prepared. 

n 

B 

B 

B 

B 

D 

B 

B 

B 

B 

Choose  my  words  with  care.  | 

a 

B 

B 

B 

B 

Do  the  opposite  of  what  is  asked. 

1 

2 

3 

4 

5 

Work  hard. 

1 

2 

3 

4 

5 

Love  order  and  regularity. 

1 

2 

3 

4 

5 

Waste  my  time. 

1 

2 

3 

4 

5 

Need  a  push  to  get  started. 

1 

2 

3 

4 

5 

Get  others  to  do  my  duties. 

1 

2 

3 

4 

5 

Like  to  tidy  up. 

i 

2 

3 

4 

5 

Find  it  difficult  to  get  down  to  work. 

i 

2 

3 

4 

5 

Excel  in  what  1  do. 

1 

2 

3 

4 

5 
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1 

2 

3 

4 

5 

Very  Inaccurate 

Inaccurate 

Neither  Inaccurate 

nor  Accurate 

Accurate 

Very  Accurate 

ID 

0 

3 

4 

5 

Often  forget  to  put  things  back  in  their 
proper  place. 

i 

2 

3 

4 

5 

i 

2 

3 

4 

5 

n 

2 

3 

4 

5 

Have  little  to  contribute. 

D 

2 

3 

4 

5 

Tell  the  truth. 

D 

2 

3 

4 

5 

Don’t  understand  things. 

1] 

2 

3 

4 

5 

Am  sure  of  my  ground. 

D 

2 

3 

4 

5 

Often  make  last-minute  plans. 

D 

2 

3 

4 

5 

Do  crazy  things. 

D 

2 

3 

4 

5 

Turn  plans  into  actions. 

D 

2 

3 

4 

5 

Misrepresent  the  facts. 

D 

0 

3 

4 

5 

Misjudge  situations. 

i 

2 

3 

4 

5 

Jump  into  things  without  thinking. 

D 

2 

3 

4 

5 

Postpone  decisions. 

D 

2 

3 

4 

5 

Listen  to  my  conscience. 

D 

2 

3 

4 

5 

i 

2 

B 

□ 

B 

Act  without  thinking. 

i 

0 

B 

□ 

b 

Leave  my  belongings  around. 

i 

0 

B 

□ 

B 

Do  more  than  what's  expected  of  me. 

D 

0 

B 

□ 

□ 

Get  chores  done  right  away. 

D 

0 

B 

□ 

0 

Do  things  according  to  a  plan. 

1 

2 

3 

4 

5 

Set  high  standards  for  myself  and  others. 

1 

2 

3 

4 

5 

Stick  to  my  chosen  path. 

1 

2 

3 

4 

5 

Am  not  bothered  by  disorder. 

1 

2 

3 

4 

5 

Complete  tasks  successfully. 

1 

2 

3 

4 

5 

Make  rash  decisions. 

1 

2 

3 

4 

5 

Want  everything  to  be  "just  right." 

1 

2 

3 

4 

5 

Am  not  bothered  by  messy  people. 

1 

2 

3 

4 

5 

Like  order. 

1 

2 

3 

4 

5 

Keep  my  promises. 

1 

2 

3 

4 

5 

Get  to  work  at  once. 

1 

2 

3 

4 

5 

Break  rules. 

1 

2 

3 

4 

5 

Avoid  mistakes. 

1 

2 

3 

4 

5 

Come  up  with  good  solutions. 

1 

2 

3 

4 

5 

Leave  a  mess  in  my  room. 

1 

2 

3 

4 

5 

Go  straight  for  the  goal. 

1 

2 

3 

4 

5 

Am  not  highly  motivated  to  succeed. 

D 

B 

B 

□ 

0 

Don't  see  the  consequences  of  things. 

i 

B 

B 

□ 

B 

Plunge  into  tasks  with  all  my  heart. 

D 

B 

B 

□ 

B 

Pay  my  bills  on  time. 

ij 

2 

T 

4 

0 

Put  little  time  and  effort  into  my  work. 

□ 

0 

B 

B 

This  scale  consists  of  a  number  of  words  that  describe  different  feelings  and  emotions.  Please 
read  each  item  and  then  place  an  X  in  the  square  that  BEST  reflects  the  extent  you  have  felt  this 
way  during  the  past  year.  Use  the  following  scale  to  record  your  answers. 


i 

2 

3 

4 

5 

Very  slightly  or  not 
at  all 

A  little 

Moderately 

Quite  a  bit 

Extremely 

interested 

1 

2 

3 

4 

5 

distressed 

1 

2 

3 

4 

5 

excited 

1 

2 

3 

4 

5 

upset 

1 

2 

3 

4 

5 

strong 

1 

2 

3 

4 

5 

guilty 

1 

2 

3 

4 

5 

scared 

1 

2 

3 

4 

5 

hostile 

1 

2 

3 

4 

5 

enthusiastic 

1 

2 

3 

4 

5 

proud 

1 

2 

3 

4 

5 

irritable 

1 

2 

3 

4 

5 

alert 

1 

2 

3 

4 

5 

ashamed 

1 

2 

3 

4 

5 

inspired 

1 

2 

3 

4 

5 

nervous 

1 

2 

3 

4 

5 

determined 

1 

2 

3 

4 

5 

attentive 

1 

2 

3 

4 

5 

jittery 

1 

2 

3 

4 

5 

active 

1 

2 

3 

4 

5 

afraid 

1 

2 

3 

4 

5 
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